Load YAML configs with environment variables interpolation
Project description
Piny
YAML configs loader with environment variables interpolation for Python.
Keep your app’s configuration in YAML file with sensitive data marked as environment variables. Put sensitive data into environment variables. Then let piny interpolate the variables on YAML loading.
Rationale
Piny combines YAML config’s readability, versioning, and environment variable’s security. Read more in the blog post.
Installation
Install using pip install -U piny. For optional validation support use one of the extra libraries in the square brackets: pip install -U 'piny[pydantic,marshmallow,trafaret]'
Usage
Set your environment variables, add them to your YAML configuration file:
db:
login: user
password: ${DB_PASSWORD}
mail:
login: user
password: ${MAIL_PASSWORD:-my_default_password}
sentry:
dsn: ${VAR_NOT_SET}
Then load your config:
from piny import YamlLoader
config = YamlLoader(path="config.yaml").load()
print(config)
# {'db': {'login': 'user', 'password': 'my_db_password'},
# 'mail': {'login': 'user', 'password': 'my_default_password'},
# 'sentry': {'dsn': None}}
You may want to discourage Bash-style envs with defaults in your configs. In such case, use a StrictMatcher:
from piny import YamlLoader, StrictMatcher
config = YamlLoader(path="config.yaml", matcher=StrictMatcher).load()
Both strict and default matchers produce None value if environment variable matched is not set in the system (and no default syntax used in the case of default matcher).
Validation
Piny supports optional data validation using third-party libraries: Marshmallow, Pydantic, Trafaret.
import marshmallow as ma
from piny import MarshmallowValidator, StrictMatcher, YamlLoader
class DBSchema(ma.Schema):
login = ma.fields.String(required=True)
password = ma.fields.String()
class ConfigSchema(ma.Schema):
db = ma.fields.Nested(DBSchema)
config = YamlLoader(
path="database.yaml",
matcher=StrictMatcher,
validator=MarshmallowValidator,
schema=MarshmallowConfig,
strict=True
).load(many=False)
Exceptions
LoadingError is thrown when something goes wrong with reading or parsing YAML-file. ValidationError is a wrapper for exceptions raised by the libraries for optional data validation. Original exception can be accessed by origin attribute. It comes in handy when you need more than just an original exception message (e.g. a dictionary of validation errors).
Both exceptions inherit from the PinyError.
Best practices
Maintain healthy security/convenience balance for your config
Mark up entity as an environment variable in your YAML if and only if it really is a secret (login/passwords, private API keys, crypto keys, certificates, or maybe DB hostname too? You decide)
When loading config file, validate your data. Piny supports a few popular data validation tools.
Store your config files in the version control system along with you app’s code.
Environment variables are set by whomever is responsible for the deployment. Modern orchestration systems like Kubernetes make it easier to keep envs secure (see Kubernetes Secrets).
Help
Explore tests directory for more examples of usage. Also take a look at the source code and its comments. Documentation is coming soon.
Fun facts
Piny is a recursive acronym for Piny Is Not YAML. Not only it’s a library name, but also a name for YAML marked up with environment variables.
Contributing
See CONTRIBUTING.rst.
CHANGELOG
v0.4.0 (2019-06-16)
Data validators support added for Pydantic, Marshmallow (#2) by @pilosus
CONTRIBUTING.rst added (#4) by @pilosus
v0.3.1 (2019-06-09)
Minor RST syntax fix in README.rst (#9) by @pilosus
v0.3.0 (2019-06-09)
README.rst extended with Rationale and Best practices sections (#5) by @pilosus
v0.2.0 (2019-06-09)
StrictMatcher added (#3) by @pilosus
v0.1.1 (2019-06-07)
CI/CD config minor tweaks
README updated
v0.1.0 (2019-06-07)
YamlLoader added
Makefile added
CI/CD minimal pipeline added
v0.0.1 (2019-06-07)
Start the project
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.