Skip to main content

build codecov PyPI version Downloads Chidata Group Twitter URL

fuzzydata

The fuzzydata Workflow Generator

The fuzzydata workflow generator enables:

  • Abstract specification of Dataframe-based Workflows
  • Generation of randomized tables and workflows
  • Loading and replay of workflows on multiple clients

Fuzzydata is currently designed to run using the following clients:

fuzzydata is designed to be extensible, you may implement your own client. Please see the existing clients in fuzzydata/clients for ways to extend the abstract Artifact, Operation and Workflow classes for your client.

Installation

Manual build/install using pip.

pip install fuzzydata

fuzzydata Does not install modin or SQLAlchemy by default, but this can be specified as an install option:

pip install fuzzydata[modin|sql|all]

Usage

Some examples of fuzzydata usage are in the examples directory. You can also run the fuzzydata command to get a list of command-line options supported in fuzzydata

$ fuzzydata --help
usage: fuzzydata [-h] [--wf_client WF_CLIENT] [--output_dir OUTPUT_DIR] [--wf_name WF_NAME]
              [--columns COLUMNS] [--rows ROWS] [--versions VERSIONS] [--bfactor BFACTOR]
              [--matfreq MATFREQ] [--npp NPP] [--log LOG] [--replay_dir REPLAY_DIR]
              [--wf_options WF_OPTIONS] [--exclude_ops EXCLUDE_OPS] [--scale_artifact SCALE_ARTIFACT]

optional arguments:
  -h, --help            show this help message and exit
  --wf_client WF_CLIENT
                        Workflow Client to be used (Default pandas). Available Workflows: pandas|modin|sql
  --output_dir OUTPUT_DIR
                        Location of Output datasets to be stored
  --wf_name WF_NAME     prefix for each workflow to be generated dir to be the path prefix for these files.
  --columns COLUMNS     Number of columns in the base version
  --rows ROWS           Number of rows in the base version
  --versions VERSIONS   Number of artifact versions to generate
  --bfactor BFACTOR     Workflow Branching factor, 0.1 is linear, 100 is star-like
  --matfreq MATFREQ     Materialization frequency, i.e. how many operations before writing out an artifact
  --log LOG             Set Logging Level
  --replay_dir REPLAY_DIR
                        Replay existing workflow in directory
  --wf_options WF_OPTIONS
                        JSON-encoded workflow engine options like sql_string or modin_engine
  --exclude_ops EXCLUDE_OPS
                        JSON-encoded list of ops to exclude e.g. ["pivot"]
  --scale_artifact SCALE_ARTIFACT
                        JSON-encoded dict of {artifact_label: new_size} to be scaled up e.g. {"artifact_0"
                        : 1000000}

Documentation

Download our paper here.

If you use fuzzydata in your research, please consider citing our paper:

@inproceedings{10.1145/3531348.3532178,
author = {Rehman, Mohammed Suhail and Elmore, Aaron},
title = {FuzzyData: A Scalable Workload Generator for Testing Dataframe Workflow Systems},
year = {2022},
isbn = {9781450393539},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3531348.3532178},
doi = {10.1145/3531348.3532178},
booktitle = {Proceedings of the 2022 Workshop on 9th International Workshop of Testing Database Systems},
pages = {17–24},
numpages = {8},
location = {Philadelphia, PA, USA},
series = {DBTest '22}
}

License

MIT License

Contributing to fuzzydata

Check out the current roadmap in docs/roadmap.md. You are always welcome to develop a new client for fuzzydata.

Contact

Suhail Rehman / ChiData Group @ Uchicago CS

Metadata

Release files for fuzzydata 0.0.11

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for fuzzydata 0.0.11
File Size Uploaded Provenance Actions
fuzzydata-0.0.11.tar.gz 29.4 kB Without provenance Details for fuzzydata-0.0.11.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for fuzzydata 0.0.11
File Interpreter ABI Platform Actions
fuzzydata-0.0.11-py3-none-any.whl Python 3 none any Details for fuzzydata-0.0.11-py3-none-any.whl

Total release size: 62.8 kB

Release files / fuzzydata-0.0.11.tar.gz

Download URL fuzzydata-0.0.11.tar.gz
Size 29.4 kB
Tags Source
SHA-256 checksum
How to use checksums
13cc0e58ae977eb1254c6bafe21fffea3a5092a8ec5bd32ee887bb12f0888e5f
BLAKE2b-256 checksum
How to use checksums
2920f9a555d44fe915589d4b021f3aefca7ad90b3a3497896d0ac0ce143685d0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.12.1

Release files / fuzzydata-0.0.11-py3-none-any.whl

Download URL fuzzydata-0.0.11-py3-none-any.whl
Size 33.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
05ac2dc55714c2f7540f4a4ea9acced0ec18e320c472b65f417fffb2e94e75c3
BLAKE2b-256 checksum
How to use checksums
939997eb7372ba415dbe382f97b37ccc8277dd90bbf16457e6102ffcafa3ebe0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.2 CPython/3.12.1

Security for fuzzydata 0.0.11

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page