gConfigs - Config and Secret parser for your Python projects.
Project description
Let me show you some code:
from gconfigs import envs, configs, secrets
HOME = envs('HOME', default='/')
DEBUG = configs.as_bool('DEBUG', default=False)
DATABASE_USER = configs('DATABASE_USER')
DATABASE_PASS = secrets('DATABASE_PASS')
>>> # envs, configs and secrets are iterables
>>> from gconfigs import envs
>>> for env in envs:
... print(env)
... print(env.key)
... print(env.value)
...
EnvironmentVariable(key='ENV_TEST', value='env-test-1')
ENV_TEST
env-test-1
...
>>> 'ENV_TEST' in envs
True
>>> envs.json()
'{"ENV_TEST": "env-test-1", "HOME": "/root", ...}'
This is experimental, so you know, use at your own risk.
Features
Python 3.6
No dependencies
Read configs from:
Installation
To install gConfigs, run this command in your terminal:
$ pip install gconfigs
$ # or
$ pipenv install gconfigs
These are the preferred methods to install gConfigs.
If you don’t have pip or pipenv installed, this Python installation guide can guide you through the process.
Usage - Basics
I will show you the basics with the built-in backends.
I’m still deciding about other backends. If you need a custom backend, it’s easy to create. Check “Advanced” section for more.
Environment Variables
So, there are good reasons to not use environment variables for your configs, but if you want / need to use, please just not use for sensitive data, like: passwords, secret keys, private tokens and stuff like that.
>>> from gconfigs import envs
# contents from ``envs`` are just data from ``os.environ``
>>> envs
<GConfigs backend=LocalEnv>
>>> envs('HOME')
'/root'
Local Mounted Configs and Secrets
Configs and secrets can be mounted as text files, read-only and in a secure location if possible, and we can read its contents. Basically the file name will be like a var / key name and its contents will be the value.
configs
For configs, gConfigs will look for mouted files at /run/configs, for example:
File Absolute Path: /run/configs/LANGUAGE_CODE File Name: LANGUAGE_CODE File Contents: en-us
from gconfigs import configs
LANGUAGE_CODE = configs('LANGUAGE_CODE')
# ...translates into:
LANGUAGE_CODE = "en-us"
Of course you can change the path that gConfigs will look for your configs. Let’s suppose your configs are mouted at /configs:
from gconfigs import configs
configs.root_dir = '/configs'
# will look for /configs/LANGUAGE_CODE
LANGUAGE_CODE = configs('LANGUAGE_CODE')
This is the simplest way to do it. Check section “Advanced” for more.
secrets
It follows the same flow as configs, so for more details go to configs.
For secrets, gConfigs will look for mouted files at /run/secrets.
from gconfigs import secrets
POSTGRES_PASSWORD = secrets('POSTGRES_PASSWORD')
# ...translates into:
POSTGRES_PASSWORD = "super-strong-password"
secrets.root_dir = '/secrets'
# will look for /secrets/POSTGRES_PASSWORD
POSTGRES_PASSWORD = secrets('POSTGRES_PASSWORD')
NOTE: If you don’t know what tools follow these workflows for configurations and secrets, you could try with Docker. Check Docker Configs and / or Docker Secrets management with Docker.
.env (dotenv) files
.env files are present not only in Python projects, for that reason many developers are familiar with, it’s just like a .ini file, but without the sections, you could say it’s a key-value store in a file.
.env files could be a good solution depending on your stack. It’s better than environment variables at least.
You could just put your configurations in a file called .env, (or whatever name you want), for example the contents of your file would be:
ROOT=/
PROJECT_NAME=gConfigs - Config and Secret parser
AUTH_MODULE=users.User
After that I’m going to save my .env file in /app/, then the full path will be /app/.env, now let’s see how to load all it’s contents in gConfigs:
from gconfigs import dotenvs
dotenvs.load_file('/app/.env')
# after that it's like using ``envs``, or ``configs``
ROOT = dotenvs('ROOT')
NAME = dotenvs('PROJECT_NAME')
AUTH = dotenvs('AUTH_MODULE')
- NOTES:
if it’s a .ini syntax it will be parsed, but it will ignore sections
duplicated keys will be overridden by the latest value
inexistent keys or empty files will raise exception
all values load as strings, use casting to convert them
didn’t like the name dotenvs? Just do: from gconfigs import dotenvs as configs
Usage - Advanced
With the basics, you are already running your projects just fine, but if you want the extra stuff of gConfigs, I’ll show you.
I’ll be using envs in the examples, but it should work for all built-in backends.
Get Your Config Value
default value
You can provide a default value, in case the backend couldn’t return the config.
>>> from gconfigs import envs
>>> envs('WHATEVER', default='/')
'/'
typed value
Generally backends will return key and value as strings, but you can return other types.
gconfigs.GConfigs.get won’t try to cast your typed value.
For example when providing a default value you could set a int:
>>> from gconfigs import envs
>>> envs('WORKERS', default=1)
1
But you must know that if your backend, in that case it’s just the LocalEnv backend, return a string value, you could create a bug in your configuration. Unless your software is prepared to deal with the number of WORKERS being a string and an integer, you could be in trouble.
What you want here is to cast your value, that you could achieve by simply converting what gConfigs return to the desired type or using some of the built-in casting methods.
casting (converting your strings to a specific type)
Most of the backends will return a string (str) as value. But sometimes you want to use a bool, int, list config.
NOTE: I choose to not do too much magic, so the cast methods implemented for gConfigs just loads the values with json.loads from the Python’s built-in json module. Therefore, it must be a valid json value, I’ll show you how:
BOOLEAN - Converting to bool
Let’s say you want DEBUG as a boolean.
>>> from gconfigs import envs
>>> envs.as_bool('DEBUG')
True
I’m not doing any magic translation of "on" => True | "yes" => True. I don’t want to introduce ambiguity, In my opinion, configurations must be straightforward and with limited variations.
LIST - Converting to list
Let’s say you have a configuration value like this:
[1, 2.1, "string-value", true]
# if you want to try in your terminal:
export CONFIG_LIST='[1, 2.1, "string-value", true]'
The value must be just JSON-like, which is very close to a list in Python. And you will be able to get a list object by doing:
>>> from gconfigs import envs
>>> envs.as_list('CONFIG-LIST')
[1, 2.1, 'string-value', True]
DICT - Converting to dict
If you have a value that is basically a JSON valid object, you may already know you can turn into a dict using json.loads.
Here is an example, if your config value is:
{"endpoint": "/", "workers": 1, "debug": true}
# if you want to try in your terminal:
export CONFIG_DICT='{"endpoint": "/", "workers": 1, "debug": true}'
>>> from gconfigs import envs
>>> envs.as_list('CONFIG-LIST')
{'endpoint': '/', 'workers': 1, 'debug': True}
Again, nothing new, no surprises, boring, no magic… as intended.
OTHER TYPES - Converting to int, float, tuple, str, set
Well let’s not reinvent the wheel, like I said before, most backends will return string by default, so if we have something like:
WORKERS="1"
WEIGHT="1.1"
MODULES='["auth", "session"]'
We could then do this:
>>> from gconfigs import envs
>>> int(envs('WORKERS'))
1
>>> float(envs('WEIGHT'))
1.1
If you want tuple or set, just get as list and then do whatever you want:
>>> from gconfigs import envs
>>> tuple(envs.as_list('MODULES'))
('auth', 'session')
>>> set(envs.as_list('MODULES'))
{'auth', 'session'}
What about strings? If you getting from your backend config values that aren’t strings, and for some of them you need to convert to str, just use the Python built-in str():
>>> from gconfigs import envs
>>> envs('AN-INT-CONFIG') # if this return an integer
1
>>> str(envs('AN-INT-CONFIG')) # just use str
'1'
Interators
>>> from gconfigs import envs
>>> list(envs) # envs is a iterator
[EnvironmentVariable(key='LANG', value='C.UTF-8'), ...]
>>> for env in envs:
... print(env)
... print(env.key)
... print(env.value)
...
EnvironmentVariable(key='ENV_TEST', value='env-test-1')
ENV_TEST
env-test-1
...
If you use an iterator once, you can’t iterate again, but if you want you can call .iterator() and get a new one:
>>> iter_envs = envs.iterator()
>>> for env in iter_envs:
... print(env.key)
...
HOME
LANG
Extra Goodies
How many configs with Python built-in len.
Config key exists with Python built-in in.
Output your key-value configs as JSON.
>>> from gconfigs import envs
>>> len(envs)
28
>>> 'HOME' in envs
True
>>> envs.json()
'{"HOME": "/root", ...}'
Beyond: from gconfigs import envs
Let’s see some stuff you can do more than just import the ready for use configs and secrets.
We have GConfigs class which takes data from one of the backends gconfigs.backends and and add fancy stuff like casting and iterator behaviour.
A backend is simply a class implementing the methods:
get(key: str): return a value given a key
keys(): return all available keys
If you know some Python, just look the gconfigs.backends.LocalEnv and you’ll see there’s no secret.
Extending a Built-in Backend
Okay let’s create a practical example of how to override the behaviour of one of our backends.
If you get your Configs and Secrets with gconfigs.configs and gconfigs.secrets, you are making use of gconfigs.LocalMountFile backend. That being said we could extend gconfigs.LocalMountFile and make it only get the configs if they are a mount point.
from gconfigs import GConfigs, LocalMountFile
import os
class MountPointConfigs(LocalMountFile):
def get(self, key, **kwargs):
file = self.root_dir / key
if os.path.ismount(file):
return super().get(key, **kwargs)
raise Exception(f"The config {key} file must be a mount point.")
# :backend: can be a callable class or a instance
# :object_type_name: it's just the name of the namedtuple you get when you
# iterate over `configs`.
configs = GConfigs(
backend=MountPointConfigs, object_type_name="MountPointConfig"
)
MY_CONFIG = configs('MY_CONFIG')
(if you use Docker Configs or Docker Secrets, you probably know that it does mount your configs / secrets in your container filesystem)
Create Your Own Backend
If you want to extend the usage of gConfigs with other backends, it’s not a hard task.
Imagine my configs are stored in Redis (a key-value store), a backend for this would look like:
class RedisBackend:
"""Redis Backend for gConfigs
NOTE: this is an example, so you probably would have to install the "redis"
python package, then connect to Redis, then you would be able to implement
``get`` and ``keys`` methods.
"""
def keys(self):
# return a iterable of all keys available
return available_keys
def get(self, key: str):
# this method receive a key (identifier of a config)
# and return its respective value
return value
gConfigs only expects you provide two methods:
- get(key: str): return a value given a key
connect to your backend
based on the key get it’s value
return the value OR raise exception if it was not possible to get the config
keep in mind that the return type it’s up to you, str makes things kinda agnostic
- keys(): return all available keys
connect to your backend
return an iterable (list, tuple, generator..) of all available keys if possible
if you don’t want or it’s not possible to implement this, just raise a NotImplementedError or a more informative exception if you like
- (Optional) load_file(filepath: str): parse file and just raise exception if fails
IMPORTANT: the method name it has to be load_file, that way gConfigs will provide a load_file that just calls the backend to load the file, check gconfigs.GConfigs.__init__ for more
read the file
parse and get keys and values
store the keys and values inside a dict if you want
then implement get and keys as described above
You could also look at the module gconfigs.backends, so you can see how the built-in backends are implemented.
What’s Next
More backends, the really fun ones
Don’t know, you tell me on Issues
History
v0.1.1 (09-04-2018)
Open sourcing the project
v0.1.2 (15-04-2018)
Mostly documentation and dev setup patches
Fix failing test on travis
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for gconfigs-0.1.2-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2578469ca4cf786e5643f98f72fe6012002c5ee13b7e027c610dd85ba20e0d8e |
|
MD5 | 7453aecb739f0d92856f2fb2b8e1d2b2 |
|
BLAKE2b-256 | 8c12fe6ff036d80c2ad005bcaa3435b47da5487e781b0d5ef3b6910efa82067a |