███╗ ███╗ ██████╗ ██████╗██╗ ██╗██████╗ ██╗██████╗ ███████╗ ████╗ ████║██╔═══██╗██╔════╝██║ ██╔╝██╔══██╗██║██╔══██╗██╔════╝ ██╔████╔██║██║ ██║██║ █████╔╝ ██████╔╝██║██████╔╝█████╗ ██║╚██╔╝██║██║ ██║██║ ██╔═██╗ ██╔═══╝ ██║██╔═══╝ ██╔══╝ ██║ ╚═╝ ██║╚██████╔╝╚██████╗██║ ██╗██║ ██║██║ ███████╗ ╚═╝ ╚═╝ ╚═════╝ ╚═════╝╚═╝ ╚═╝╚═╝ ╚═╝╚═╝ ╚══════╝
MockPipe
There’s a lot of sample databases out there and lots of ways to generate some dummy data (i.e. faker, which this project uses), but i couldn’t find much in the way of dynamically generating realistic data that could be used to generate some scenarios that one might actually find coming out of a operational systems CDC feed. This is an attampt to create a utility/library that can be used to setup some .
From a yaml config a set of sample tables can be defined, using dummy default values for any newly generated rows along with a set of actions that can be performed with a certain frequency.
The dummy values actually invoke the Faker library to generate somewhat realistic entries, along with support for other data types that may refer to existing values within the table or other tables so that relationships can be maintained.
Data is persisted onto a duckdb database so the outputs can be persisted between executions and support any other analysis/queries you may want to do.
Features
Dynamic Data Generation: Generate sample tables from a YAML configuration, using dummy default values for newly generated rows.
Faker Integration: Leverage the Faker library to create realistic entries.
Relationship Maintenance: Support for data types that refer to existing values within the same table or other tables, ensuring relationships are preserved.
Action Frequency: Define a set of actions to be performed with a certain frequency.
Persistence: Data is persisted in a DuckDB database, allowing outputs to be saved between executions and enabling further analysis or queries.
Installation
To install Mockpipe, you can use pip:
pip install mockpipe
Basic Usage
import mockpipe
# Define your YAML configuration
yaml_config = """
tables:
- name: users
columns:
- name: id
type: integer
primary_key: true
- name: name
type: string
faker: name
- name: email
type: string
faker: email
actions:
- table: users
action: insert
frequency: 1.0
# Initialize Mockpipe with the configuration
mp = mockpipe.Mockpipe(yaml_config)
Command line Usage
Usage: mockpipe [OPTIONS]
Options:
--config_create generate a sample config file
--config PATH path to yaml config file
--steps INTEGER Number of steps to execute initially
--run-time INTEGER Time to run the mockpipe process in seconds
--dry-run Validate the config file and exit without running anything
--verbose Enable verbose logging
--version Show the version and exit.
--help Show this message and exit.
Config Specification
Top Level Keys
key |
value type |
allowed values |
default value |
sample |
explanation |
|---|---|---|---|---|---|
db_path |
path |
any |
mockpipe.db |
sample.db |
path of duckdb db |
delete_behaviour |
string |
[soft, hard] |
soft |
soft |
whether deleted records will be marked as deleted with ‘D’ or actually hard deleted in the persisted db |
inter_action_delay |
float |
0.0 -> |
0.5 |
0.1 |
delay between each action |
output |
table |
output format |
|||
full_load |
table |
N/A |
periodic full-table snapshot export, alongside the incremental change stream. See ‘Full Load’ |
Output
key |
value type |
allowed values |
default value |
sample |
explanation |
|---|---|---|---|---|---|
format |
string |
[json, csv, parquet] |
json |
json |
file format output |
path |
path |
any |
extract |
extract |
folder path for output |
Full Load
If the full_load key is present, every full_load.frequency recorded changes (across all tables combined), mockpipe exports a full snapshot of each table’s current rows in addition to the normal incremental export for the change that just happened. This is useful for downstream consumers that periodically want a fresh full baseline rather than only ever seeing incremental changes.
key |
value type |
allowed values |
default value |
sample |
explanation |
|---|---|---|---|---|---|
include_deletes |
bool |
any |
false |
true |
whether soft-deleted rows are included in the full snapshot |
frequency |
int |
1 -> |
100 |
50 |
how many recorded changes between each full snapshot export |
Note: include_deletes: true is incompatible with delete_behaviour: hard, since hard-deleted rows no longer exist to include.
Tables
key |
value type |
allowed values |
default value |
sample |
explanation |
|---|---|---|---|---|---|
name |
string |
any |
N/A |
employees |
table name used. Also used for output |
fields |
table |
List of fields in table |
|||
actions |
table |
List of actions within table |
Fields
key |
value type |
allowed values |
default value |
sample |
explanation |
Note |
|---|---|---|---|---|---|---|
name |
string |
any |
N/A |
order_date |
table name used. Also used for output |
|
type |
string |
[string, int, float, boolean] |
N/A |
string |
List of fields in table |
|
value |
string |
[increment, static(*), table_random(), fake.*] |
N/A |
fake.date_between |
List of actions within table |
See ‘Field Value Usage’ |
arugments |
list |
any |
N/A |
|
Arguments to pass to faker functions |
See ‘Field Value Usage’ |
Actions
key |
value type |
allowed values |
default value |
sample |
explanation |
Note |
|---|---|---|---|---|---|---|
name |
string |
any |
N/A |
update_order_status |
name of action |
|
field |
string |
any |
N/A |
order_status |
field which gets updated |
|
action |
string |
[create, delete, set] |
N/A |
set |
type of action to perform |
|
value |
string |
[increment, static(*), table_random(), fake.*] |
N/A |
fake.random_element |
value to set field to |
|
arguments |
list |
any |
N/A |
(‘pending’, ‘completed’, ‘shipped’, ‘delivered’) |
if using faker, arguments to pass |
|
frequency |
float |
0->1 |
N/A |
0.25 |
relative frequency of action |
|
where_condition |
string |
<table>.<value> == <condition> |
N/A |
products.product_id == table_random(products, product_id, 0) |
where condition to limit which rows in table to apply action to |
See where condition |
action_condition |
string |
EFFECT_ONLY |
N/A |
EFFECT_ONLY |
used to specify if the action is only ever to be invoked by another action (i.e., an effect) |
|
effect |
string |
<table>.<action>(<target_col>=<source_col>, …) |
N/A |
product.product_count(order_id=order_id) |
After the specified action is executed, another action can be invoked, passing values onwards to the next action |
See Effect |
effect_count |
[int, string] |
0->max(int), inherit |
N/A |
inherit |
if effect is set, how many times to invoke the next effect |
See Effect |
effect_count_random |
string |
<min>,<max> |
N/A |
1,5 |
if effect is set, how many times to invoke the next effect |
See Effect |
Field Values
type |
increment |
|---|---|
explanation |
Will only wok for integer fields. It acts as you’d expect, incrementing the value by 1 for each new row generated and selecting a random value from the specified table respectively. |
syntax |
increment |
examples |
increment |
type |
static |
|---|---|
explanation |
Will set a static value on each new row generated. This can be any value you want, but it will be the same for each row generated. |
syntax |
static(<value>) |
examples |
static(false), static(100), static('pending') |
type |
table_random |
|---|---|
explanation |
Will select a random value from the specified table for each new row generated. Note, will only select non-deleted rows. It’s important to set a default value in case the table is empty. |
syntax |
table_random(<table_name>, <column_name>, <default_value>) |
examples |
table_random(products, product_id, 0) |
type |
fake.* |
|---|---|
explanation |
Will generate a value using the faker library. The arguments key can be used to pass arguments to the faker function. |
syntax |
fake.<faker_function> |
examples |
fake.company |
Effects
The effect is used to specify that after the specified action is executed, another action can be invoked, passing values onwards to the next action. This can be useful for chaining actions together to create one to one, one to many relationships, you can also specify how many times to invoke the next
effect:
explanation |
Which action to invoke after the current action is executed. |
syntax |
<table>.<action>(<target_col>=<source_col>, <target_col=<source_col>, ...) |
example |
effect: product.product_count(order_id=order_id) |
effect_count:
explanation |
If the effect is set, how many times to invoke the next effect. Note, can not be used with effect_count_random. |
syntax |
<int> |
example |
1 |
effect_count_random:
explanation |
If the effect is set, how many times to invoke the next effect. Note, can not be used with effect_count. |
syntax |
<min>,<max> |
example |
1,5 |
action_condition:
Used to specify if the action is only ever to be invoked by another action (i.e., an effect).
explanation |
Used to specify if the action is only ever to be invoked by another action (i.e., an effect). |
syntax |
EFFECT_ONLY |
example |
EFFECT_ONLY |
Where Condition
explanation |
The where condition is used to limit which rows in the table an action is applied to. It can be set to a filter, i.e. where status==’pending’ or it can perform a lookup to another table to get the value to filter on. |
syntax |
<table>.<value> == / != / >= / <= / > / < <condition> |
table_random condition syntax |
table_random(<table_name>, <column_name>, <default_value>) |
static syntax |
static(<value>) |
table_random example |
products.product_id == table_random(orders, product_id, 0) |
static example |
products.product_id == static(1) |
Future Enhancements
simplify action usage and allow for duckdb functions
move from raw SQL string concatenation to parameterized queries
additional exporters (e.g. direct-to-Postgres/S3)
Contributing
Contributions are welcome, Please open an issue or submit a pull request on GitHub.
License
This project is licensed under the MIT License. See the LICENSE file for details.
Acknowledgements
[Faker](https://github.com/joke2k/faker) - For generating realistic dummy data.
[DuckDB](https://duckdb.org/) - For data persistence and analysis.
Metadata
Release files for mockpipe 0.0.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| mockpipe-0.0.7.tar.gz | 42.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| mockpipe-0.0.7-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 71.7 kB
Release files / mockpipe-0.0.7.tar.gz
| Download URL | mockpipe-0.0.7.tar.gz |
|---|---|
| Size | 42.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b66018ec68d69ff3f82c5691fab3ec06da4eb01fd75fd2255c13e6c8c0a4f977
|
|
BLAKE2b-256 checksum How to use checksums |
8121cabc5ed70ad9f395a0ffeb5aa83dfba07fc8829e1ec474322858692d4254
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 31, 2026.
Transparency logRelease files / mockpipe-0.0.7-py3-none-any.whl
| Download URL | mockpipe-0.0.7-py3-none-any.whl |
|---|---|
| Size | 29.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7223427d1b0c61e4cb9d22330306bbd1e20e040f459a57f91fcee52d98e82d8f
|
|
BLAKE2b-256 checksum How to use checksums |
3e2dfe186f50f04c93b88803c3472611f3ebbea900654d7868c94a0a259b47f5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 31, 2026.
Transparency log