Knockoff Factory

A library for generating mock data and creating database fixtures that can be used for unit testing.

Table of content

• Installation
• Changelog
• Documentation
• Unit Tests
• Future Work
• Legacy YAML Based CLI

Installation

From PyPi:

pip install knockoff

# to install with PyMySQL 
pip install knockoff[mysql]
# Note: Other MySql DBAPI's can be used if installed and dialect provided in connection url

From GitHub:

pip install git+https://github.com/Nike-Inc/knockoff-factory#egg=knockoff

# to install with PyMySQL 
pip install git+https://github.com/Nike-Inc/knockoff-factory#egg=knockoff[mysql]
# Note: Other MySql DBAPI's can be used if installed and dialect provided in connection url

Changelog

See the changelog for a history of notable changes to knockoff.

Documentation

We are working on adding more documentation and examples!

• Knockoff SDK
  • KnockoffTable
  • KnockoffDB
• TempDatabaseService
• Knockoff CLI
• Unit Testing Example: Sample App

Unit Tests

Prerequisites

• docker
• poetry (curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python)

Some of the unit tests depend on a database instance. Knockoff will create ephemeral databases within that instance and clean them up when tests have completed. By default it will attempt to connect to an existing instance at postgresql://postgres@localhost:5432/postgres and will create and destroy databases per test. This postgres location can be overridden with the KNOCKOFF_TEST_DB_URI environment variable.

If no external postgres instance is available for testing, but postgresql is installed, the TEST_USE_EXTERNAL_DB environment variable can be set to 0. The fixtures will then rely on the testing.postgresql library to create ephemeral postgres instances per fixture.

If postgres is not available, dependent tests can be disabled with the following: export TEST_POSTGRES_ENABLED=0.

Some tests also depend on a MySql database instance. These tests can be disabled with the following: export TEST_MYSQL_ENABLED=0.

Create the database instance using docker:

# Run postgres instance 
docker run --rm  --name pg-docker -e POSTGRES_HOST_AUTH_METHOD=trust -d -p 5432:5432  postgres:11.9

# Run mysql instance
docker run --name mysql-docker -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -p 3306:3306 -d mysql:8.0.26

Install poetry:

# the -E flag is so we can run the mysql unit tests with the PyMySql DBAPI
poetry install -E mysql

Run unit test:

poetry run pytest

Future work

• Further documentation and examples for SDK
• Add yaml based configuration for SDK
• Make extensible generic output for KnockffDB.insert (csv, parquet, etc)
• Enable append option for KnockoffDB.insert
• Autodiscover and populate all tables by using reflection and building dependency graph with foreign key relationships
• Parallelize execution of dag. (e.g. https://ipython.org/ipython-doc/stable/parallel/dag_dependencies.html)