Skip to main content

Library for generating and bootstrapping mock data

Project description

Knockoff Factory

codecov Test PyPi Release Docker Build License Python Versions Docker Image Size Python Wheel

A library for generating mock data and creating database fixtures that can be used for unit testing.

Table of content

Installation

From PyPi:

pip install knockoff

# to install with PyMySQL 
pip install knockoff[mysql]
# Note: Other MySql DBAPI's can be used if installed and dialect provided in connection url

From GitHub:

pip install git+https://github.com/Nike-Inc/knockoff-factory#egg=knockoff

# to install with PyMySQL 
pip install git+https://github.com/Nike-Inc/knockoff-factory#egg=knockoff[mysql]
# Note: Other MySql DBAPI's can be used if installed and dialect provided in connection url

Changelog

See the changelog for a history of notable changes to knockoff.

Documentation

We are working on adding more documentation and examples!

Unit Tests

Prerequisites

  • docker
  • poetry (curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python)

Some of the unit tests depend on a database instance. Knockoff will create ephemeral databases within that instance and clean them up when tests have completed. By default it will attempt to connect to an existing instance at postgresql://postgres@localhost:5432/postgres and will create and destroy databases per test. This postgres location can be overridden with the KNOCKOFF_TEST_DB_URI environment variable.

If no external postgres instance is available for testing, but postgresql is installed, the TEST_USE_EXTERNAL_DB environment variable can be set to 0. The fixtures will then rely on the testing.postgresql library to create ephemeral postgres instances per fixture.

If postgres is not available, dependent tests can be disabled with the following: export TEST_POSTGRES_ENABLED=0.

Some tests also depend on a MySql database instance. These tests can be disabled with the following: export TEST_MYSQL_ENABLED=0.

Create the database instance using docker:

# Run postgres instance 
docker run --rm  --name pg-docker -e POSTGRES_HOST_AUTH_METHOD=trust -d -p 5432:5432  postgres:11.9

# Run mysql instance
docker run --name mysql-docker -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -p 3306:3306 -d mysql:8.0.26

Install poetry:

# the -E flag is so we can run the mysql unit tests with the PyMySql DBAPI
poetry install -E mysql

Run unit test:

poetry run pytest

Future work

  • Further documentation and examples for SDK
  • Add yaml based configuration for SDK
  • Make extensible generic output for KnockffDB.insert (csv, parquet, etc)
  • Enable append option for KnockoffDB.insert
  • Autodiscover and populate all tables by using reflection and building dependency graph with foreign key relationships
  • Parallelize execution of dag. (e.g. https://ipython.org/ipython-doc/stable/parallel/dag_dependencies.html)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

knockoff-4.4.1.tar.gz (36.2 kB view hashes)

Uploaded Source

Built Distribution

knockoff-4.4.1-py3-none-any.whl (57.2 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page