tailucas-pylib 0.1.4

Common Python utility modules

About The Project

Overview

This package was created by factoring out many reusable code artifacts from my various projects over a number of years. Since this work was not a part of a group effort, the test coverage is predictably abysmal :raised_eyebrow: and Python documentation notably absent :expressionless:. For each of my projects, which derive from a common Docker application found here, this package provides some specific and relatively powerful features to enable rapid offshoots for new ideas. Here is what functionality this package provides:

• aws.init.py: If AWS environment variables are set, global boto and boto3 session objects are instantiated.
• aws.leader: Leader-election mechanism that makes use of DynamoDB that relies on conditional updates.
• aws.metrics: CloudWatch metric helper.
• aws.swf: Client-side interface to botoflow with interfaces specific to a number of prior projects. Examples of this use are in respective swf branches of some projects. This mechanism has been since replaced with message passing via RabbitMQ.
• init.py: A lot of module bootstrap with a pitiful attempt to support unit testing. At application startup, a choice is made between logging to a configured logger or to the terminal if one exists, to support a scenario where the application is run interactively. A standard application configuration file is loaded with a Python ConfigParser and then a credentials client is loaded based on 1Password. The choice of 1Password is motivated by their secrets automation features and because I happened to use their service already. I have been tempted to try Bitwarden also but there has not been a need yet to create this abstraction. Next, the Sentry client is loaded with whatever extras the given application might specify. Then Python builtins are abused to store some globals for use throughout the package and application.
• app: A few of my applications use ZeroMQ for "lockless" IPCs between Python threads and so it became apparent that some helpers were needed to support common patterns. ZmqRelay is one such example where an application thread can be used to receive some message on which processing is done, and then forward it to another ZeroMQ channel. The main thread logic is contained with a context manager which handles failures and shutdown gracefully.
• app.bluetooth: Bluetooth helper functions that make use of hcitool l2ping.
• app.data: Simple utility to make a common IPC or network payload using MessagePack.
• app.datetime: Date and timestamp manipulation with time zone normalization.
• app.handler: A useful Python context manager to handle a variety of failure conditions.
• app.leader: A crude leader election mechanism using RabbitMQ. I would not recommend this without some kind of sensible testing of actual consensus. This was built mainly to experiment with the idea of some form of cooperative leader election.
• app.process: Simple signal handler with interaction with application shutdown tracking.
• app.rabbit: A fairly good example of a RabbitMQ application framework with internal IPC handoff to ZeroMQ channels for inter-thread communication.
• app.threads: A useful thread nanny with shutdown debugging and metric support.
• app.zmq: Useful helper functions for a ZeroMQ enabled application.

Handy stand-alone tools:

• config_interpol: By making creative abuse of Python's ConfigParser, this tool designed for the command-line will take a configuration file with variables that are automatically substituted with with either an overlay configuration or environment variables by the same name and output the interpolated configuration. A good example of this tool being used is here.
• cred_tool: Useful to fetch an item from 1Password.
• yaml_interpol: A script useful to generate docker-compose YAML output from templates. A crude example is here which fetches application configuration from a 1Password vault. Note that application runtime secrets are only in memory and loaded in init.py. I happen to use 1Password to also store application key-value pairs for use in docker-compose templates.

Package Structure

A crude attempt to visualize the package structure with pyreverse.

(back to top)

Built With

Technologies that help make this package useful:

Also:

• Cronitor
• MessagePack

• Botoflow

(back to top)

Getting Started

Here is some detail about the intended use of this package.

Prerequisites

A Python project or runtime environment. Since this project is already initialized with Poetry dependency management, I recommend that you continue to use it. Beyond the Python dependencies defined in the Poetry configuration, the package init carries hardcoded dependencies on Sentry and 1Password in order to function. Unless you want these and are effectively extending my base project, you're likely better off forking this package and cutting out what you do not need.

Installation

This project is intended to be used as a Git submodule because it is not yet structured or built to be vended from the Python Package Index. One of my companion projects illustrates how easily the project can be added to the dependency closure of a project using Poetry which also greatly diminished the need to publish to online package indexes.

(back to top)

Usage

I have various projects that use this tool chain. For example, my Base Project which can be run stand-alone but also serves as my Docker base image from which other projects are derived.

(back to top)

License

Distributed under the MIT License. See LICENSE for more information.

(back to top)

Acknowledgments

• Template on which this README is based

(back to top)