Skip to main content
███╗   ███╗ ██████╗  ██████╗██╗  ██╗██████╗ ██╗██████╗ ███████╗
████╗ ████║██╔═══██╗██╔════╝██║ ██╔╝██╔══██╗██║██╔══██╗██╔════╝
██╔████╔██║██║   ██║██║     █████╔╝ ██████╔╝██║██████╔╝█████╗
██║╚██╔╝██║██║   ██║██║     ██╔═██╗ ██╔═══╝ ██║██╔═══╝ ██╔══╝
██║ ╚═╝ ██║╚██████╔╝╚██████╗██║  ██╗██║     ██║██║     ███████╗
╚═╝     ╚═╝ ╚═════╝  ╚═════╝╚═╝  ╚═╝╚═╝     ╚═╝╚═╝     ╚══════╝

Latest version released on PyPI Build status of the master branch Package license


MockPipe

There’s a lot of sample databases out there and lots of ways to generate some dummy data (i.e. faker, which this project uses), but i couldn’t find much in the way of dynamically generating realistic data that could be used to generate some scenarios that one might actually find coming out of a operational systems CDC feed. This is an attampt to create a utility/library that can be used to setup some .

From a yaml config a set of sample tables can be defined, using dummy default values for any newly generated rows along with a set of actions that can be performed with a certain frequency.

The dummy values actually invoke the Faker library to generate somewhat realistic entries, along with support for other data types that may refer to existing values within the table or other tables so that relationships can be maintained.

Data is persisted onto a duckdb database so the outputs can be persisted between executions and support any other analysis/queries you may want to do.

Features

  • Dynamic Data Generation: Generate sample tables from a YAML configuration, using dummy default values for newly generated rows.

  • Faker Integration: Leverage the Faker library to create realistic entries.

  • Relationship Maintenance: Support for data types that refer to existing values within the same table or other tables, ensuring relationships are preserved.

  • Action Frequency: Define a set of actions to be performed with a certain frequency.

  • Persistence: Data is persisted in a DuckDB database, allowing outputs to be saved between executions and enabling further analysis or queries.

Installation

To install Mockpipe, you can use pip:

pip install mockpipe

Basic Usage

import mockpipe

# Define your YAML configuration
yaml_config = """
tables:
  - name: users
    columns:
      - name: id
        type: integer
        primary_key: true
      - name: name
        type: string
        faker: name
      - name: email
        type: string
        faker: email
actions:
  - table: users
    action: insert
    frequency: 1.0

# Initialize Mockpipe with the configuration
mp = mockpipe.Mockpipe(yaml_config)

Command line Usage

Usage: mockpipe [OPTIONS]

Options:
  --config_create     generate a sample config file
  --config PATH       path to yaml config file
  --steps INTEGER     Number of steps to execute initially
  --run-time INTEGER  Time to run the mockpipe process in seconds
  --version           Show the version and exit.
  --help              Show this message and exit.

Config Specification

Top Level Keys

key

value type

allowed values

default value

sample

explanation

db_path

path

any

mockpipe.db

sample.db

path of duckdb db

delete_behaviour

string

[soft, hard]

soft

soft

whether deleted records will be marked as deleted with ‘D’ or actually hard deleted in the persisted db

inter_action_delay

float

0.0 ->

0.5

0.1

delay between each action

output

table

output format

Output

key

value type

allowed values

default value

sample

explanation

format

string

[json, csv]

json

json

file format output

path

path

any

extract

extract

folder path for output

Tables

key

value type

allowed values

default value

sample

explanation

name

string

any

N/A

employees

table name used. Also used for output

fields

table

List of fields in table

actions

table

List of actions within table

Fields

key

value type

allowed values

default value

sample

explanation

Note

name

string

any

N/A

order_date

table name used. Also used for output

type

string

[string, int, float, boolean]

N/A

string

List of fields in table

value

string

[increment, static(*), table_random(), fake.*]

N/A

fake.date_between

List of actions within table

See ‘Field Value Usage’

arugments

list

any

N/A

  • “-1y”

  • “today”

Arguments to pass to faker functions

See ‘Field Value Usage’

Actions

key

value type

allowed values

default value

sample

explanation

Note

name

string

any

N/A

update_order_status

name of action

field

string

any

N/A

order_status

field which gets updated

action

string

[create, delete, set]

N/A

set

type of action to perform

value

string

[increment, static(*), table_random(), fake.*]

N/A

fake.random_element

value to set field to

arguments

list

any

N/A

(‘pending’, ‘completed’, ‘shipped’, ‘delivered’)

if using faker, arguments to pass

frequency

float

0->1

N/A

0.25

relative frequency of action

where_condition

string

<table>.<value> == <condition>

N/A

products.product_id == table_random(products, product_id, 0)

where condition to limit which rows in table to apply action to

See where condition

action_condition

string

EFFECT_ONLY

N/A

EFFECT_ONLY

used to specify if the action is only ever to be invoked by another action (i.e., an effect)

effect

string

<table>.<action>(<target_col>=<source_col>, …)

N/A

product.product_count(order_id=order_id)

After the specified action is executed, another action can be invoked, passing values onwards to the next action

See Effect

effect_count

[int, string]

0->max(int), inherit

N/A

inherit

if effect is set, how many times to invoke the next effect

See Effect

effect_count_random

string

<min>,<max>

N/A

1,5

if effect is set, how many times to invoke the next effect

See Effect

Field Values

type

increment

explanation

Will only wok for integer fields. It acts as you’d expect, incrementing the value by 1 for each new row generated and selecting a random value from the specified table respectively.

syntax

increment

examples

increment

type

static

explanation

Will set a static value on each new row generated. This can be any value you want, but it will be the same for each row generated.

syntax

static(<value>)

examples

static(false), static(100), static('pending')

type

table_random

explanation

Will select a random value from the specified table for each new row generated. Note, will only select non-deleted rows. It’s important to set a default value in case the table is empty.

syntax

table_random(<table_name>, <column_name>, <default_value>)

examples

table_random(products, product_id, 0)

type

fake.*

explanation

Will generate a value using the faker library. The arguments key can be used to pass arguments to the faker function.

syntax

fake.<faker_function>

examples

fake.company

Effects

The effect is used to specify that after the specified action is executed, another action can be invoked, passing values onwards to the next action. This can be useful for chaining actions together to create one to one, one to many relationships, you can also specify how many times to invoke the next

effect:

explanation

Which action to invoke after the current action is executed.

syntax

<table>.<action>(<target_col>=<source_col>, <target_col=<source_col>, ...)

example

effect: product.product_count(order_id=order_id)

effect_count:

explanation

If the effect is set, how many times to invoke the next effect. Note, can not be used with effect_count_random.

syntax

<int>

example

1

effect_count_random:

explanation

If the effect is set, how many times to invoke the next effect. Note, can not be used with effect_count.

syntax

<min>,<max>

example

1,5

action_condition:

Used to specify if the action is only ever to be invoked by another action (i.e., an effect).

explanation

Used to specify if the action is only ever to be invoked by another action (i.e., an effect).

syntax

EFFECT_ONLY

example

EFFECT_ONLY

Where Condition

explanation

The where condition is used to limit which rows in the table an action is applied to. It can be set to a filter, i.e. where status==’pending’ or it can perform a lookup to another table to get the value to filter on.

syntax

<table>.<value> == / != / >= / <= / > / < <condition>

table_random condition syntax

table_random(<table_name>, <column_name>, <default_value>)

static syntax

static(<value>)

table_random example

products.product_id == table_random(orders, product_id, 0)

static example

products.product_id == static(1)

Future Enhancements

  • improved yaml config validation

  • improved logging

  • increased test coverage

  • simplyfy action usage and allow for duckdb functions

  • better typing support

Contributing

Contributions are welcome, Please open an issue or submit a pull request on GitHub.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Acknowledgements

Metadata

Release files for mockpipe 0.0.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mockpipe 0.0.5
File Size Uploaded
mockpipe-0.0.5.tar.gz 34.3 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for mockpipe 0.0.5
File Interpreter ABI Platform
mockpipe-0.0.5-py3-none-any.whl Python 3 none any Details

Total release size: 60.1 kB

Release files / mockpipe-0.0.5.tar.gz

Download URL mockpipe-0.0.5.tar.gz
Size 34.3 kB
Tags Source
SHA-256 checksum
How to use checksums
27c2e1c467f5b9a635d0b9bdf697eaa23855c007ab5b1fdfbdd3d15f56e157b1
BLAKE2b-256 checksum
How to use checksums
679b585af7972bcdfe8ae9ab9d7b9a8856da11e4f3ce7e28b65d568d6765b776
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jul 31, 2026.

Transparency log

Release files / mockpipe-0.0.5-py3-none-any.whl

Download URL mockpipe-0.0.5-py3-none-any.whl
Size 25.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
348638db06534bbeeb956ea962165630300ccc51d6ffcdd16d0ace26d8a18047
BLAKE2b-256 checksum
How to use checksums
acf00d1473e5d329b72e7af0981f118ca770651754a5379067b0a25c4bb7ded8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jul 31, 2026.

Transparency log

Release history Release notifications | RSS feed

0.0.10

2 release files

0.0.9

2 release files

0.0.8

2 release files

0.0.7

2 release files

0.0.6

2 release files

This release

0.0.5 This release

2 release files

0.0.4

2 release files

0.0.3

2 release files

0.0.2

2 release files

0.0.1

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page