Skip to main content
Datafast logo

Generate diverse synthetic datasets with LLMs, from a pipeline you can read.

PyPI version Python versions License Docs GitHub stars

Documentation · Quickstart · Cookbook · Examples · API reference

Datafast is a Python library for developers and ML engineers who need a dataset for an LLM project and do not have one yet.

A plain script that loops over prompts gets hard to control. You must write the nested loops that cover each combination of inputs. Without checkpoints, a crash halfway means you pay for the same LLM calls again. Without metadata, you cannot tell which input and which model made each record.

With datafast, you declare the dimensions that your dataset must cover, connect steps into a pipeline, and run it. The pipeline writes the dataset to JSONL, CSV, Parquet or the Hugging Face Hub.

Install

pip install datafast

Datafast needs Python 3.10 or newer. Extras add Parquet, the Hugging Face Hub and Langfuse tracing: see Installation.

Quick example

This example generates exam questions and answers for 3 science topics at 2 levels, beginner and advanced. The result is 6 example records.

Set the key for your provider, here OpenAI:

export OPENAI_API_KEY="sk-..."
from datafast import LLMStep, Seed, Sink, openai

pipeline = (
    Seed.product(
        Seed.values("topic", ["photosynthesis", "plate tectonics", "vaccines"]),
        Seed.values("level", ["beginner", "advanced"]),
    )
    >> LLMStep(
        prompt=(
            "Write one {level} exam question about {topic}, with its answer. "
            "Return JSON with fields question and answer."
        ),
        input_columns=["topic", "level"],
        output_columns=["question", "answer"],
        parse_mode="json",
        model=openai(),
    )
    >> Sink.jsonl("questions.jsonl")
)

pipeline.run()

The pipeline has 3 steps:

  1. Seed.product makes 1 record for each combination of the values in its dimensions: 3 topics × 2 levels = 6 records.
  2. LLMStep makes 1 LLM call for each record. With parse_mode="json", it splits the response into the question and answer columns.
  3. Sink.jsonl writes the records to questions.jsonl.

Each line of the file is 1 record. The seed columns stay beside the generated columns, and _model tells you which model wrote the record:

{
  "topic": "photosynthesis",
  "level": "beginner",
  "question": "What process do plants use to convert light into chemical energy?",
  "answer": "Photosynthesis, which occurs in the chloroplasts.",
  "_model": "gpt-6-luna"
}

The Quickstart explains each step in more detail. Key concepts and the API reference list all the steps.

Use cases

  • Make evaluation data for an LLM project before you have real user data.
  • Add records to an existing dataset to cover more cases.
  • Test an LLM application on realistic inputs before you ship it.
  • Make datasets to fine-tune LLMs and agents.
  • Distill a large model into a smaller one: the large model writes the training data.

Why datafast

  • You declare the coverage. You give the values of each dimension, and Seed.product makes all the combinations. You do not write nested loops.
  • A long run can resume after a crash. Set checkpoint_dir to save progress, then restart with the same directory and resume=True. See Checkpoints and resume for recovery limits.
  • You can change the provider in 1 line. The same provider factories work for OpenAI, Anthropic, Gemini, Mistral, OpenRouter, Ollama and all OpenAI-compatible servers such as vLLM and llama.cpp. See Served models.
  • Datafast finds mistakes before you pay. Pipeline.compile() checks the step order and all column references before the first LLM call.
  • Each record shows its origin. The seed values and the model id stay in the record, so you can filter or audit the dataset later.

Cookbook

Each cookbook is a complete pipeline that makes a real dataset. The script is in examples/scripts/, and the page explains each step.

Cookbook Dataset Script
Text classification Trail reports from hikers in 2 languages, with labels 45
Personas Personas made from real articles 43
Domain text corpus Spacecraft engineering text in 2 languages 44
Preference pairs Chosen and rejected answers with scores, for RLHF 42

The examples page lists all 45 scripts. Scripts 01 to 14 do not use an API key.

Contributing

Bug reports, ideas and pull requests are welcome. Open an issue or read the contributing guide.

License

Datafast is released under the Apache 2.0 license.

If datafast helps you, give the repo a star. It helps other people find it.

Metadata

Release files for datafast 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datafast 1.0.0
File Size Uploaded Provenance Actions
datafast-1.0.0.tar.gz 219.1 kB Without provenance Details for datafast-1.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datafast 1.0.0
File Interpreter ABI Platform Actions
datafast-1.0.0-py3-none-any.whl Python 3 none any Details for datafast-1.0.0-py3-none-any.whl

Total release size: 322.7 kB

Release files / datafast-1.0.0.tar.gz

Download URL datafast-1.0.0.tar.gz
Size 219.1 kB
Tags Source
SHA-256 checksum
How to use checksums
67225057cd9ebb5c2912d89fcc58d8705c58f19b40f364c9ae8a412ae055dbee
BLAKE2b-256 checksum
How to use checksums
4c3f2cd8b336092cf45892ee8ecfceff4b19cf2ccf4990e74e8617f60cc3ddf7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Release files / datafast-1.0.0-py3-none-any.whl

Download URL datafast-1.0.0-py3-none-any.whl
Size 103.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4dcd17611c9e8057d36757ad2abfc11f0ebc4fbc54e9fe97e00c6fea8be745ec
BLAKE2b-256 checksum
How to use checksums
29b181b178e21b82876f3821f0e456b19534ca913dca0fc0153168ff3c1cdcb4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Security for datafast 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page