mockpipe 0.0.10

MockPipe

There’s a lot of sample databases out there and lots of ways to generate some dummy data (i.e. faker, which this project uses), but i couldn’t find much in the way of dynamically generating realistic data that could be used to generate some scenarios that one might actually find coming out of a operational systems CDC feed. This is an attampt to create a utility/library that can be used to setup some .

From a yaml config a set of sample tables can be defined, using dummy default values for any newly generated rows along with a set of actions that can be performed with a certain frequency.

The dummy values actually invoke the Faker library to generate somewhat realistic entries, along with support for other data types that may refer to existing values within the table or other tables so that relationships can be maintained.

Data is persisted onto a duckdb database so the outputs can be persisted between executions and support any other analysis/queries you may want to do.

Features

• Dynamic Data Generation: Generate sample tables from a YAML configuration, using dummy default values for newly generated rows.

• Faker Integration: Leverage the Faker library to create realistic entries.

• Relationship Maintenance: Support for data types that refer to existing values within the same table or other tables, ensuring relationships are preserved.

• Action Frequency: Define a set of actions to be performed with a certain frequency.

• Persistence: Data is persisted in a DuckDB database, allowing outputs to be saved between executions and enabling further analysis or queries.

Installation

To install Mockpipe, you can use pip:

pip install mockpipe

Basic Usage

import mockpipe

# Define your YAML configuration
yaml_config = """
tables:
  - name: users
    fields:
      - name: id
        type: int
        value: increment
        is_pk: true
      - name: name
        type: string
        value: fake.name
      - name: email
        type: string
        value: fake.email
    actions:
      - name: create
        action: create
        frequency: 1.0
"""

# Initialize MockPipe with the configuration
mp = mockpipe.MockPipe(yaml_config)

# Run for 10 steps
for _ in range(10):
    mp.step()

Command line Usage

Usage: mockpipe [OPTIONS]

Options:
  --config-create              generate a sample config file
  --config PATH                path to yaml config file
  --steps INTEGER               Number of steps to execute initially
  --run-time INTEGER           Time to run the mockpipe process in seconds
  --dry-run                    Validate the config file and exit without running anything
  --output-format TEXT         Override the config file's output.format (e.g. json, csv, parquet, webhook, kafka)
  --output-path TEXT           Override the config file's output.path
  --output-url TEXT            Override the config file's output.url (used by the webhook format)
  --output-topic TEXT          Override the config file's output.topic (used by the kafka format)
  --output-bootstrap-servers TEXT
                                Override the config file's output.bootstrap_servers (used by the kafka format)
  --verbose                    Enable verbose logging
  --version                    Show the version and exit.
  --help                       Show this message and exit.

Config Specification

Top Level Keys

Output

Note: without batch_size, mockpipe writes one output file per changed row, which can produce a large number of small files over a long run. Buffered rows are also written out when the process stops (or via MockPipe.flush_exports() if you only ever call step() directly).

Note: the webhook format POSTs each exported batch as JSON ({"table": <table_name>, "rows": [...]}) to output.url, instead of writing a file - useful for testing a downstream consumer directly. output.path is unused for this format.

Note: the kafka format publishes each exported batch as a JSON message (same shape as webhook) to output.topic on the brokers listed in output.bootstrap_servers (comma-separated for more than one), instead of writing a file. The producer connection is kept open and reused for the life of the run, and flushed on stop()/flush_exports(). If the topic doesn’t already exist, you may see a one-off “Topic not found” warning logged while the broker auto-creates it - this is expected and the message is still delivered.

Reproducible Runs (seed)

Set the top-level seed key to a fixed integer to make a run reproducible - useful for turning a flaky/failing test run into something you can reliably re-run. With the same seed and the same config, Faker-generated field values and action/table selection will be identical between runs.

Note: table_random lookups rely on DuckDB’s own USING SAMPLE clause, which DuckDB does not guarantee is bit-for-bit reproducible even with a fixed seed (an upstream engine limitation) - mockpipe still seeds it, but treat that part as best-effort rather than guaranteed.

Full Load

If the full_load key is present, every full_load.frequency recorded changes (across all tables combined), mockpipe exports a full snapshot of each table’s current rows in addition to the normal incremental export for the change that just happened. This is useful for downstream consumers that periodically want a fresh full baseline rather than only ever seeing incremental changes.

Note: include_deletes: true is incompatible with delete_behaviour: hard, since hard-deleted rows no longer exist to include.

mockpipe also creates its own _mockpipe_metadata table in your db_path database (alongside the tables you define). It tracks how many changes have been recorded so far, so that full_load’s schedule resumes correctly if you re-open the same db_path in a later run instead of restarting from zero. You can safely ignore this table - it isn’t exported and isn’t part of the data mockpipe generates for you.

Tables

Fields

Actions

Field Values

Note: for fake.random_element (or any other faker method accepting a set of choices), passing a dict-literal string as the argument instead of a tuple gives each option a weighted probability instead of a uniform chance, e.g.:

value: fake.random_element
arguments:
  - "{'pending': 0.05, 'shipped': 0.05, 'delivered': 0.9}"

The values don’t need to sum to 1 - they’re relative weights, not strict probabilities.

Effects

The effect is used to specify that after the specified action is executed, another action can be invoked, passing values onwards to the next action. This can be useful for chaining actions together to create one to one, one to many relationships, you can also specify how many times to invoke the next

effect:

effect_count:

effect_count_random:

action_condition:

Used to specify if the action is only ever to be invoked by another action (i.e., an effect).

Where Condition

Future Enhancements

• simplify action usage and allow for duckdb functions

• move from raw SQL string concatenation to parameterized queries

• additional exporters (e.g. direct-to-Postgres/S3)

Contributing

Contributions are welcome, Please open an issue or submit a pull request on GitHub.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Acknowledgements

• [Faker](https://github.com/joke2k/faker) - For generating realistic dummy data.

• [DuckDB](https://duckdb.org/) - For data persistence and analysis.