Python Devops Template Tool

Command-line interface for setting up a Python project based on a dev-ops template.

• Sample project for this template (including detailed documentation).
• Cookiecutter for this template.
• Also check out audreyr/cookiecutter-pypackage for additional Python package templates.

Features

This command-line interface supports the creation and the management of a Python dev-ops template that provides:

• testing and deployment in a multi-stage Docker environment,
• packaging with setuptools,
• code analysis with pylint, bandit, pytest and coverage,
• code quality monitoring with SonarQube.

The dev-ops pipeline is mostly implemented in a Makefile and a Dockerfile which are independent of your Python code. A SonarQube server is started with docker-compose.

Creation and management of the template:

• create a new instance of the template (very similar to Cookiecutter),
• manage an existing instance/project by adding template components as the project evolves,
• generate a Cookiecutter template (see lrothack/cookiecutter-pydevops),
• packages the template code in a Python distributions, e.g., binary wheel package,
• configure the template with boolean command-line flags or in interactive mode,
• resolves author information automatically with git config.

Optional template components:

• MongoDB
• MlFlow (with PostgreSQL and MinIO backends)

These components have been developed in jkortner/ml-ops.

Installation

Install the latest version from pypi.org:

pip install -U devopstemplate

From source:

# Obtain sources
git clone --recurse-submodules https://github.com/lrothack/dev-ops-admin.git

# Install and activate virtual environment
cd dev-ops-admin
python3 -m venv venv
source venv/bin/activate

# Build package
make dist

The binary wheel package is located in the dist directory and can be installed with pip.

Create and manage projects

After installation, the executable devopstemplate is available. It provides the sub-commands:

• create
• manage
• cookiecutter

An overview of the functionalities is shown on the help screens:

devopstemplate --help
devopstemplate <sub-command> --help

The working directory is always the root directory of your project, for example:

mkdir sampleproject
cd sampleproject
devopstemplate create

Using the dev-ops template

After creating a new project or after switching to the project directory:

• Set up a virtual environment for your project and activate it (requires Python >= 3.6).
• Run make help in order to get an overview of the targets provided by Makefile. Note: Running make is only supported from project directory.
• Run make install-dev in order to install the package (and all dependencies) in development mode.
• Run make lint in order to run code analysis with pylint and bandit.
• Run make test in order to run unit tests with pytest and coverage.
• Run make dist in order to build a Python package (binary and source).
• Make sure you have Docker installed and the Docker daemon is running. Allocate at least 4GB RAM in the Docker resource configuration.
• Run docker-compose -p sonarqube -f sonarqube/docker-compose.yml up -d in order to start a SonarQube server. Configure your server through its web interface and obtain an authentication token. The SonarQube URL can be configured through the Makefile variable SONARURL. The authentication token can be stored in the local file .sonartoken.
• Run make sonar in order to run sonar-scanner and report results to your SonarQube server.
• Run make docker-build in order to analyze, test, package and deploy in a multi-stage Docker build. Analysis results and test results are shown after the build.

Advanced configurations can be made in the configuration sections of Makefile. See lrothack/dev-ops for more information.

Additional components

• mongodb
• mlflow

Additional components can be installed when creating a new project or with the manage command at a later time (replace <component> with a component from the list above, also see devopstemplate manage --help):

cd sampleproject
devopstemplate manage --add-<component>

Start the corresponding Docker containers with docker-compose:

docker-compose -p <component> -f <component>/docker-compose.yml up -d

Also check out the README file in the <component> directory and run the sample script.