configirl 0.0.9

Centralized Config Management Tool.

---

Welcome to configirl Documentation

• What is configirl

• What problem does configirl solve

• Quick Start

• Additional Features

• Use Case - Java Web App Project with AWS, Serverless, Infrastructure as Code

• Other Godies

• Install

What is configirl

configirl is a single script, pure python, no dependencies, python2.7, 3.4+ compatible, drop in ready python library to help you manage complex config value logic. This devops solution applies to ANY PROJECT, ANY PROGRAMMING LANGUAGE.

from configirl import ConfigClass, Constant, Deriable

class Config(object):
    PROJECT_NAME = Constant()
    PROJECT_NAME_SLUG = Deriable()

    @PROJECT_NAME_SLUG.getter
    def get_project_name_slug(self):
        return self.PROJECT_NAME.get_value().replace("_", "-")

config = Config(PROJECT_NAME="my_project")

What problem does configirl solve

Devops Engineer has to deal with lots of config and parameters everyday. Some config value are just a constant value, like a integer and a string. Some config value can be derived from other config values, sometimes event requires the context.

There are lots of Devops tools available in the community, such as:

• Shell Script for command line tool, automation

• Jenkins groovy for CI/CD

• Cloudformation for Infrastructure as Code

• Terraform for Infrastructure as Code

• …

They all using different language and different syntax. The method of managing config value in different tools varies very much! If you have to manage a list of config values, and you are using multiple devops tools in the same project. Allow those tools talk to each other is NOT EASY at all. And the effort to manage config value in certain tools might be very difficult (like CloudFormation).

configirl provides a solution to manage complex logic for config values in an elegant way. Since Python is easy to learn and it is full featured programming language, you got the perfect balance of simplicity and flexibility. To integrate with any Devops tools, you just reference the value from the finalized config JSON file.

Quick Start

1. Copy configirl.__init__.py to your Devops workspace directory, and rename it as configirl.py. That is for drop in ready.

2. Create a config-raw.json file put the following content:

{
    "PROJECT_NAME": "my_project",
    "STAGE": "dev"
}

3. Create a config.py file, put the following content. Since it is Python2.7, 3.4+ compatible, pure Python, no dependencies, it works everywhere.

from configirl import ConfigClass, Constant, Derivable

class Config(object):
    CONFIG_DIR = "your-devops-workspace-dir"

    PROJECT_NAME = Constant()
    PROJECT_NAME_SLUG = Derivable()

    @PROJECT_NAME_SLUG.getter
    def get_project_name_slug(self):
        return self.PROJECT_NAME.get_value().replace("_", "-")

    @PROJECT_NAME_SLUG.validator
    def check_project_name_slug(self, value):
        if "_" in value:
            raise ValueError("you can't use `_` in slugifie name!")

    STAGE = Constant()

    ENVIRONMENT_NAME = Derivable()

    @PROJECT_NAME_SLUG.getter
    def get_environment_name(self):
        return "{}-{}".format(
            self.PROJECT_NAME_SLUG.get_value(),
            self.STAGE.get_value(),
        )

config = Config()
config.update_from_raw_json_file()
config.dump_shell_script_json_config_file()
config.dump_cloudformation_json_config_file()

# you can call more custom dump method here
# depends on what other devops tools you are using

4. Everytime you call python config.py then the ground truth config value in config-raw.json will be parsed. and two more config-final-for-shell-script.json, config-final-for-cloudformation.json will be create. Then you can just reference value from thos xxx-final-xxx.json file.

// content of config-final-for-shell-script.json
{
    "PROJECT_NAME": "my_project",
    "PROJECT_NAME_SLUG": "my-project",
    "STAGE": "dev",
    "ENVIRONMENT_NAME": "my-project-dev"
}

// content of config-final-for-cloudformation.json
{
    "ProjectName": "my_project",
    "ProjectNameSlug": "my-project",
    "Stage": "dev",
    "EnvironmentName": "my-project-dev"
}

Additional Features

1. you can custom your validator.

from configirl import ConfigClass, Constant, Derivable

class Config(object):
    PROJECT_NAME = Constant()
    PROJECT_NAME_SLUG = Derivable()

    @PROJECT_NAME_SLUG.getter
    def get_project_name_slug(self):
        return self.PROJECT_NAME.get_value().replace("_", "-")

    @PROJECT_NAME_SLUG.validator
    def check_project_name_slug(self, value):
        if "_" in value:
            raise ValueError("you can't use `_` in slugifie name!")

2. you can inherit your Config Class.

from configirl import ConfigClass, Constant, Derivable

class Config1(object):
    PROJECT_NAME = Constant()

class Config2(Config1):
    PROJECT_NAME_SLUG = Derivable()

    @PROJECT_NAME_SLUG.getter
    def get_project_name_slug(self):
        return self.PROJECT_NAME.get_value().replace("_", "-")

    @PROJECT_NAME_SLUG.validator
    def check_project_name_slug(self, value):
        if "_" in value:
            raise ValueError("you can't use `_` in slugifie name!")

class Config(Config2):
    CONFIG_DIR = "your-devops-workspace-dir"

config = Config()
... do what every you need

Use Case - Java Web App Project with AWS, Serverless, Infrastructure as Code

In this example, we are designing the devops solution for a complex Web App, the app logic is written in JAVA Sprint, the application code is deployed to Amazon Web Service via Cloudformation, lots of microservices are deployed to AWS Lambda and AWS ApiGateway with Serverless framework, and use CircleCI to automate the test, build, deployment.

Suppose your project name is MyWebApp, and it has multiple deployment stage dev, test, prod, in other word, it will be deployed to three Environment. And the environment name MyWebApp-dev/test/prod will be used as a prefix name almost everywhere in your Java Code, Cloudformation Code, CICD Code. And you DONT want to manage the config value like PROJECT_NAME and STAGE everywhere in Java Code, Cloudformation Code, CICD Code.

If you don’t want to create the devops scripts manually in the following instruction, you can just copy the entire devops-example directory from https://github.com/MacHu-GWU/configirl-project/tree/master/devops-example to your local machine.

1. Centralize Your Config Definition

The easiest way to use configirl is to copy the configirl.__init__.py file to your Devops workspace directory, and rename it as configirl.py. It is drop in ready and no dependencies, it runs any Mac or Linux Machine.

Create a config.py file next to configirl.py it is the centralized place to manage your config logic, put the following code in config.py, it defines two major constant variables PROJECT_NAME and STAGE, and two derivable variables PROJECT_NAME_SLUG and ENVIRONMENT_NAME:

# -*- coding: utf-8 -*-
# content of config.py

"""
defines the constant and derivable config value.
"""

import os
from configirl import ConfigClass, Constant, Derivable


class Config(ConfigClass):
    CONFIG_DIR = os.path.dirname(__file__)

    PROJECT_NAME = Constant()  # example "MyWebApp"
    PROJECT_NAME_SLUG = Derivable()

    @PROJECT_NAME_SLUG.getter
    def get_PROJECT_NAME_SLUG(self):
        return self.PROJECT_NAME.get_value().replace("_", "-")

    @PROJECT_NAME_SLUG.validator
    def check_PROJECT_NAME_SLUG(self, value):
        if "_" in value:
            raise ValueError("you can't use `_` in slugifie name!")

    STAGE = Constant()  # example "dev"

    ENVIRONMENT_NAME = Derivable()

    @ENVIRONMENT_NAME.getter
    def get_ENVIRONMENT_NAME(self):
        return "{}-{}".format(
            self.PROJECT_NAME_SLUG.get_value(),
            self.STAGE.get_value(),
        )

    APP_PUBLIC_URL = Derivable()
    @APP_PUBLIC_URL.getter

2. Create the Config Data for Different Enviornment.

Create three config files ./01-config-dev.json, ./01-config-test.json, ./01-config-prod.json, and put the following contect in corresponding files {"STAGE": "dev"}, {"STAGE": "test"}, {"STAGE": "prod"}.

Create a config file ./00-config-shared.json and put the following content {"PROJECT_NAME": "MyWebApp"}.

For different deployment stages, they may share common config values, those information goes to ./00-config-shared.json file.

For environment dependent config values, they goes to different config files.

3. Write your Config initiation Scripts.

Where the config values been load from may varies in different environment.

• On local development, the configs may sit on your local computer.

• On CI/CD environment, the configs may comes from Git, and sensitive information may be stored in secure storage like AWS Secret Manager.

• On EC2 App server, the configs may comes from Environment Variables.

You can create a config_init.py scripts that tells computer to load config values from different place in different situations.

# -*- coding: utf-8 -*-
# content of config_init.py

"""
initialize the config object, it reads common config value from the
``00-config-shared.json`` file, and read environment specified value from the
``config-raw.json`` file.

Suppose that:

- on local development, you load all values from your local file.
- on CI/CD environment, you load non-sensitive values from Git repo, load
    sensitive values from AWS Secret Manager. Because you don't 100% trust your
    CI/CD provider.
- on EC2 App server, you load non-sensitive values from Git repo, load
    sensitive values from Environment Variable. Because the servers locates
    at secure environment.
"""

import os, json
from config import Config

conf = Config()

path_shared_config_file = os.path.join(os.path.dirname(__file__), "00-config-shared.json")
path_shared_secrets_config_file = os.path.join(os.path.dirname(__file__), "00-config-shared-secrets.json")


# load non sensitive values
conf.update(json.loads(open(path_shared_config_file, "rb").read().decode("utf-8")))

# load environment specified values
conf.update_from_raw_json_file() # load environment specific values

# load sensitive values
if conf.is_aws_ec2_runtime():
    conf.update_from_env_var(prefix="APP_CONFIG_")
elif conf.is_ci_runtime():
    def read_sensitive_value_from_aws_secret_manager():
        return dict()
    conf.update(read_sensitive_value_from_aws_secret_manager())
else:
    conf.update(json.loads(open(path_shared_secrets_config_file, "rb").read().decode("utf-8")))

# dump other derivable values for other system to use
if conf.is_ci_runtime(): # allow other system like terraform to use those value for deployment
    conf.dump_shell_script_json_config_file()
    conf.dump_terraform_json_config_file()

Other Godies

• pysecret Allows to easily and securely load from and write to file, environment variable, aws secret manager.

Install

configirl is released on PyPI, so all you need is:

$ pip install configirl

To upgrade to latest version:

$ pip install --upgrade configirl