Skip to main content
Join the official 2020 Python Developers SurveyStart the survey!

Quick and Painless Python Config File Tool

Project description

Build Status Coverage Status PyPI version

QuickFig - Quick and Painless Config Parser for Python

A lightweight schema-supporting config parser for projects.

QuickFig Release Notes

  • 0.3.0

    • Add support setting parameters via environment variables
    • Fix bug causing issues finding definitions in section mode
  • 0.2.0

    • Add support for Python 2.7
  • 0.1.1

    • Initial version

Quick Intro to QuickFig

QuickFig is a quick and easy to use library for configuring your application using a Yaml file. QuickFig allows you to set up a loose schema with default values and data types that reduces error checking and config processing code in your app.

I have written something like this for almost every project I have done and it always take longer than it should, so I decided to write it as a library and share it.

Features

  • Set a loose schema for the configuration. While parameters do not require a schema defined, it helps as it will take care of the conversion for it.

    • Example Schema:

        schema = {
                  'runtime.debug': {
                      'desc': "DEBUG Mode',
                      'type': 'bool', 
                      'default': 'false', 
                      'env': [ 'MYAPP_DEBUG', 'DEBUG']},
                  'myapp.float': {
                      'type': 'float', 
                      'default': 1.2},
                  'myapp.str': {
                      'type': 'str',
                      'default': "string value"}
        }
      
        config = QuickFig(definition=schema)
      

      The above defines parameter 'runtime.debug' which is a boolan that defaults to False if not defined, you can set it by setting env variables MYAPP_DEBUG or DEBUG, but if both are set, value in MYAPP_DEBUG will win. It also defines "myapp.float" and myapp.str as a float and a string respectively

    • A Schema for a parameter can define:

      • desc - Description of the parameter
      • type - Parameter Type
        • Pre-Defined Data Types (Additional types may be added dynamically as needed):
          • str - Strings (default if not set)
          • bool - Boolean
          • int - Integer
          • float - Floating point
          • list - A List of values
          • dict - A key-value dictionary
      • default - Default value for parameter
      • env - Environment variable(s) that can be used if parameter is not set
        • If at least one env variable is set and parameter is not defined in the config file, the first environment variable that is defined will be used.
    • When Schema is not defined, QuickFig will take a best guest at parameter type based on current value.

  • Read a YAML configuration file

    • The file format can be of any depth and may include parameters with or without a schema

    • Example: (Using same schema)

      • Config File: (config.yaml) myapp: str: My String float: 9.2

      • Code:

          ## Env variable DEBUG is set to "yes"
          config = QuickFig(definition=schema)
        
          # No data yet, only schema default
          assert config.myapp.str == "string value"
        
          #Load config from file
          config.quickfig_load_from_file('config.yaml')
        
          # Data from config file:
          assert config.myapp.str == "My String"
        
          # Data From env variable DEBUG
          # Note that it got converted to a boolean from string
          assert config.runtime.debug == True
        
  • Set parameter overrides at runtime that set the value regardless of env variables or values in config files. This is handy if you want to allow, for example, user to override a config file parameter from command line

    • Example:

        #Note that value can be a string or a boolean
        overrides={'runtime.debug': 'yes'}
        config = QuickFig(definition=schema, overrides=overrides)
      
        assert config.runtime.debug = True
      
  • Store all of this configuration in an object that allows:

    • Access to read/get any value using dotter notation:

      • Example:

          config = QuickFig(definitions=schema)
        
          # No data yet, only schema default
          assert config.myapp.str == "string value"
        
          #set the value
          config.set("myapp.str", "new_value")
        
          # Get the value using dict-like get(key, default_value)
          assert config.get("myapp.str", "default_value") == "new_value"
        
    • Access parameters using property notation (setting this way is not supported):

      • Example:

          config = QuickFig(definitions=schema)
        
          # create a sub-config for section `myapp`
          myapp_config = config.section('myapp')
        
          #set the value
          myapp_config.set("str", "new_value")
        
          # Get the value using full config object
          assert config.myapp.str == "new_value"
        
          # Get the value using full sub-config object
          assert myapp_config.str == "new_value"
        
    • Creation of a filtered sub-config object that allows access to child nodes without using full path:

      • Example:

          config = QuickFig(definitions=schema)
        
          # No data yet, only schema default
          assert config.myapp.str == "string value"
        
          #set the value
          config.set("myapp.str", "new_value")
        
          # Get the value using property notation
          assert config.myapp.str == "new_value"
        

Usage

Concepts

Key concept in QuickFig is that while the config file can be netsted, it can always be flattended using the dotted notation - so that something like this:

    myapp:
        section_1:
           parameter: value
        section_2:
           parameter: another value

can also be represented as:

    myapp.section_1.parameter: value
    myapp.section_2.parameter: value

While the main config file can be in either format - the schema and overrides use the dotted notation

The goal is to make the config as seamless as possible to reduce validation and management code in your app.

Examples:

Full Example:

(This example exists in examples directory)

  • Config file: (full_example.conf)

     # Config file for full_example
      app:
        component1:
          enabled: true
          host: yahoo.com
        component2:
          enabled: false
          delay: 1.0
    
  • Code: (full_example.py)

      #!/usr/bin/env python3
      """
      Full Example of using QuickFig
      """
      from argparse import ArgumentParser, Action, ArgumentError
      from argparse import RawDescriptionHelpFormatter
      import os
      import sys
    
      from quickfig import QuickFig
      import yaml
    
      __version__ = 1.0
      __updated__ = ""
    
      SCHEMA_YAML = '''
      debug:
        desc: General Debug Flag
        type: bool
        default: false
        env:
          - APP_DEBUG
          - DEBUG
    
      app.component1.enabled:
        desc: Component 1 Enabled Flag
        type: bool
        default: false
    
      app.component1.host:
        desc: Component 1 Hostname
        type: str
        default: google.com
    
      app.component1.port:
        desc: Component 1 Port Number
        type: int
        default: 443
    
      app.component2.enabled:
        desc: Component 2 Enabled Flag
        type: bool
        default: no
    
      app.component2.delay:
        desc: Delay in seconds for component 2 to startup
        type: float
        default: 0.5
      '''
    
      SCHEMA = yaml.safe_load(SCHEMA_YAML)
    
    
      class StoreNameValuePair(Action):
          ''' Action to store an section.option = value pair into the  '''
    
          def __call__(self, parser, namespace, values, option_string=None):
              for value in values:
                  try:
                      (key, value) = value.split("=", 2)
                  except ValueError:
                      raise ArgumentError(self,
                                          "Could not parse argument %s as section.option=value format" % value)
                  config = getattr(namespace, self.dest) or {}
                  config[key] = value
                  setattr(namespace, self.dest, config)
    
    
      def sep(text=""):
          ''' Print a separator '''
          print('{:^60}'.format("-" * 60))
          print('---- {:^50} ----'.format(text))
          print('{:^60}'.format("-" * 60))
    
    
      def component1(config, debug):
          ''' Run component 1 '''
          print("Starting component 1")
          print("Connecting to: %s:%s" % (config.host, config.port))
          if debug:
              print("Component 1 Config:\n%s" % config)
    
    
      def component2(config, debug):
          ''' Run component 2 '''
          print("Starting component 2")
          print("Delay is: %s" % config.delay)
          if debug:
              print("Component 2 Config:\n%s" % config)
    
    
      def run():
          program_version = "v%s" % __version__
          program_build_date = str(__updated__)
          program_version_message = '%%(prog)s %s (%s)' % (
              program_version, program_build_date)
    
          parser = ArgumentParser()
          parser.add_argument("-D", dest="debug", action="store_true",
                              help="set debug mode [default: %(default)s]")
    
          parser.add_argument("-P", dest="options", metavar="option=value",
                              action=StoreNameValuePair, nargs="+",
                              help="Override config parameters, use option=value syntax")
    
          parser.add_argument('-V', '--version', action='version',
                              version=program_version_message)
    
          # Process arguments
          args = parser.parse_args()
    
          # Determine overrides based on cli args
          overrides = {}
          if args.options:
              overrides.update(args.options)
          if args.debug:
              overrides['debug'] = True
    
          # Create our config
          config = QuickFig(definitions=SCHEMA, overrides=overrides)
    
          # load the config
          config.quickfig_load_from_file("full_example.conf")
          sep("Starting...")
    
          print("Debug Mode is: %s" % ("on" if config.debug else "off"))
    
          if config.debug:
              print("Total Config: \n%s" % config)
    
          # run component1 if enabled
          if config.app.component1.enabled:
              sep("Component 1")
              # Run component 1 and give it only the config it needs
              component1(config.section('app.component1'), config.debug)
    
          # run component2 if enabled
          if config.app.component2.enabled:
              sep("Component 2")
              # Run component 1 and give it only the config it needs
              component2(config.section('app.component2'), config.debug)
          sep("Finished")
    
    
      if __name__ == "__main__":
          run()
    
  • Sample Runs:

    • Basic (no arguments)

        ./full_example.py
        ------------------------------------------------------------
        ----                    Starting...                     ----
        ------------------------------------------------------------
        Debug Mode is: off
        ------------------------------------------------------------
        ----                    Component 1                     ----
        ------------------------------------------------------------
        Starting component 1
        Connecting to: yahoo.com:443
        ------------------------------------------------------------
        ----                      Finished                      ----
        ------------------------------------------------------------
      

      As per config, debug is off, component 1 is enabled, component 2 is not

    • Override component 2 enable from command line (-P app.component2.enabled=yes)

        ./full_example.py -P app.component2.enabled=yes
        ------------------------------------------------------------
        ----                    Starting...                     ----
        ------------------------------------------------------------
        Debug Mode is: off
        ------------------------------------------------------------
        ----                    Component 1                     ----
        ------------------------------------------------------------
        Starting component 1
        Connecting to: yahoo.com:443
        ------------------------------------------------------------
        ----                    Component 2                     ----
        ------------------------------------------------------------
        Starting component 2
        Delay is: 1.0
        ------------------------------------------------------------
        ----                      Finished                      ----
        ------------------------------------------------------------
      
    • Enable Debug Mode (-D)

       ./full_example.py -D | sed "s/^/            /"
       ------------------------------------------------------------
       ----                    Starting...                     ----
       ------------------------------------------------------------
       Debug Mode is: on
       Total Config: 
       #QuickFig Config
      
       # Component 1 Enabled Flag (Default: 'False')
       app.component1.enabled = True
      
       # Component 1 Hostname (Default: 'google.com')
       app.component1.host = yahoo.com
      
       # Component 1 Port Number (Default: '443')
       app.component1.port = 443
      
       # Delay in seconds for component 2 to startup (Default: '0.5')
       app.component2.delay = 1.0
      
       # Component 2 Enabled Flag (Default: 'False')
       app.component2.enabled = False
      
       # General Debug Flag (Default: 'False')
       debug = True
      
       #End QuickFig Config
      
       ------------------------------------------------------------
       ----                    Component 1                     ----
       ------------------------------------------------------------
       Starting component 1
       Connecting to: yahoo.com:443
       Component 1 Config:
       #QuickFig Config
       #
       # Path: app.component1
       #
      
       # Component 1 Enabled Flag (Default: 'False')
       enabled = True
      
       # Component 1 Hostname (Default: 'google.com')
       host = yahoo.com
      
       # Component 1 Port Number (Default: '443')
       port = 443
      
       #End QuickFig Config
      
       ------------------------------------------------------------
       ----                      Finished                      ----
       ------------------------------------------------------------
      
      • Enable debug mode via env variable

          ------------------------------------------------------------
          ----                    Starting...                     ----
          ------------------------------------------------------------
          Debug Mode is: on
          Total Config: 
          #QuickFig Config
        
          # Component 1 Enabled Flag (Default: 'False')
          app.component1.enabled = True
        
          # Component 1 Hostname (Default: 'google.com')
          app.component1.host = yahoo.com
        
          # Component 1 Port Number (Default: '443')
          app.component1.port = 443
        
          # Delay in seconds for component 2 to startup (Default: '0.5')
          app.component2.delay = 1.0
        
          # Component 2 Enabled Flag (Default: 'False')
          app.component2.enabled = False
        
          # General Debug Flag (Default: 'False')
          debug = True
        
          #End QuickFig Config
        
          ------------------------------------------------------------
          ----                    Component 1                     ----
          ------------------------------------------------------------
          Starting component 1
          Connecting to: yahoo.com:443
          Component 1 Config:
          #QuickFig Config
          #
          # Path: app.component1
          #
        
          # Component 1 Enabled Flag (Default: 'False')
          enabled = True
        
          # Component 1 Hostname (Default: 'google.com')
          host = yahoo.com
        
          # Component 1 Port Number (Default: '443')
          port = 443
        
          #End QuickFig Config
        
          ------------------------------------------------------------
          ----                      Finished                      ----
          ------------------------------------------------------------
        

API:

Main Class: QuickFig()

This is the main class when using QuickFig.

QuickFig() Constructor
  • Keyword arguments:
    • definitions - schema as a dictionary
      • definitions should be flat(dotted notation) dictionary of key to definition dict
    • config - config as a python dictionary object
      • Dictionary may be nested or flat using dots
    • overrides - overrides as a dictionary object
      • overrides should be flat dictionary of key to value
    • resolver - an optional instance of class ConfDataTypeResolver().
      • If not specified, uses default resolver.
      • Main reason to specify is to add custom type resolution without affecting default resolver
QuickFig() Instance Methods
  • quickfig_load(config) - load configuration provided by parameter config

  • quickfig_load_from_file(filename, warn=False) - load configuration from file whose name is provided by filename parameter. If warn is true, throw a warning into the log if file not present or cannot read.

  • section(name) - get a filtered version of the config with name as prefix. Returns instance of QuickFigNode() class

  • set(key, value) - set parameter specified by key to value. key is a dotted notation representation of a parameter name.

  • get(key, default_value=None) - get parameter specified by key(dotted notation). if not set, return default_value, if specified. Note, default value specified by schema will have priority over default_value

  • get_data_type(key, test_value="") - get data type for parameter specified by key. If parameter is not set and has no schema, use value of test_value to guess at the data type. Returns object of class QuickFigDataType()

  • get_definition(key, test_value="", default_dtype=None)) - get definition for parameter specified bykey. Usetest_valueto guess at data type if no definition exists and parameter is not set. ifdefault_dtype` is specified, use that data type instead of guessing.

Class: QuickFigNode()

This class appears similar to QuickFig, but is not instanciated directly. Instead it is created via section(name) method on either QuickFig() or QuickFigNode() instances)

QuickFigNode() Instance Methods
  • section(name) - get a filtered version of the config with name as prefix. name is combined with current prefix, if one is already defined. Returns an instance of QuickFigNode() class
  • set(key, value) - set parameter specified by key to value. key is a dotted notation representation of a parameter name.
  • get(key, default_value=None) - get parameter specified by key(dotted notation). if not set, return default_value, if specified. Note, default value specified by schema will have priority over default_value
  • get_data_type(key, test_value="") - get data type for parameter specified by key. If parameter is not set and has no schema, use value of test_value to guess at the data type. Returns object of class QuickFigDataType()
  • get_definition(key, test_value="", default_dtype=None)) - get definition for parameter specified bykey. Usetest_valueto guess at data type if no definition exists and parameter is not set. ifdefault_dtype` is specified, use that data type instead of guessing.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for QuickFig, version 0.3.0
Filename, size File type Python version Upload date Hashes
Filename, size QuickFig-0.3.0-py2-none-any.whl (19.8 kB) File type Wheel Python version py2 Upload date Hashes View
Filename, size QuickFig-0.3.0-py3-none-any.whl (19.8 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size QuickFig-0.3.0.tar.gz (19.7 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page