Skip to main content
Datafast logo

Generate high-quality and diverse synthetic data with LLMs.

PyPI version Python versions License Docs GitHub stars

Documentation · Quickstart · Cookbook · Examples · API reference

Datafast is a Python library for generating synthetic datasets with LLMs.

You assemble building blocks for data operations into a custom pipeline that generates a synthetic dataset matching your project's needs. It works with open-weight and proprietary models.

Start with seeds or the dimensions your dataset must cover. Connect LLMs and data processing steps for filtering, transformations, and annotations, then run the pipeline. Save the dataset locally or push it to the Hugging Face Hub.

Install

pip install datafast

Datafast needs Python 3.11 or newer. Extras add Parquet, the Hugging Face Hub and Langfuse tracing: see Installation.

Quick example

This example generates exam questions and answers for 3 science topics at 2 levels, beginner and advanced. The result is 6 example records.

Set the key for your provider, here OpenAI:

export OPENAI_API_KEY="sk-..."
from datafast import LLMStep, Seed, Sink, openai

pipeline = (
    Seed.product(
        Seed.values("topic", ["photosynthesis", "plate tectonics", "vaccines"]),
        Seed.values("level", ["beginner", "advanced"]),
    )
    >> LLMStep(
        prompt=(
            "Write one {level} exam question about {topic}, with its answer. "
            "Return JSON with fields question and answer."
        ),
        input_columns=["topic", "level"],
        output_columns=["question", "answer"],
        parse_mode="json",
        model=openai(),
    )
    >> Sink.jsonl("questions.jsonl")
)

pipeline.run()

The pipeline has 3 steps:

  1. Seed.product makes 1 record for each combination of the values in its dimensions: 3 topics × 2 levels = 6 records.
  2. LLMStep makes 1 LLM call for each record. With parse_mode="json", it splits the response into the question and answer columns.
  3. Sink.jsonl writes the records to questions.jsonl.

Each line of the file is 1 record. The seed columns stay beside the generated columns, and _model tells you which model wrote the record:

{
  "topic": "photosynthesis",
  "level": "beginner",
  "question": "What process do plants use to convert light into chemical energy?",
  "answer": "Photosynthesis, which occurs in the chloroplasts.",
  "_model": "gpt-6-luna"
}

The Quickstart explains each step in more detail. Key concepts and the API reference list all the steps.

Use cases

  • Make evaluation data for an LLM project before you have real user data.
  • Add records to an existing dataset to cover more cases.
  • Test an LLM application on realistic inputs before you ship it.
  • Make datasets to fine-tune LLMs and agents.
  • Distill a large model into a smaller one: the large model writes the training data.

Why datafast

  • You declare the coverage. You give the values of each dimension, and Seed.product makes all the combinations. You do not write nested loops.
  • A long run can resume after a crash. Set checkpoint_dir to save progress, then restart with the same directory and resume=True. See Checkpoints and resume for recovery limits.
  • You can change the provider in 1 line. The same provider factories work for OpenAI, Anthropic, Gemini, Mistral, OpenRouter, Ollama and all OpenAI-compatible servers such as vLLM and llama.cpp. See Served models.
  • Datafast finds mistakes before you pay. Pipeline.compile() checks the step order and all column references before the first LLM call.
  • Each record shows its origin. The seed values and the model id stay in the record, so you can filter or audit the dataset later.

Cookbook

Each cookbook is a complete pipeline that makes a real dataset. The script is in examples/scripts/, and the page explains each step.

Cookbook Dataset Script
Text classification Trail reports from hikers in 2 languages, with labels 45
Personas Personas made from real articles 43
Domain text corpus Spacecraft engineering text in 2 languages 44
Preference pairs Chosen and rejected answers with scores, for RLHF 42

The examples page lists all 45 scripts. Scripts 01 to 14 do not use an API key.

Contributing

Bug reports, ideas and pull requests are welcome. Open an issue or read the contributing guide.

License

Datafast is released under the Apache 2.0 license.

If datafast helps you, give the repo a star. It helps other people find it.

Metadata

Release files for datafast 1.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datafast 1.0.1
File Size Uploaded Provenance Actions
datafast-1.0.1.tar.gz 219.8 kB Without provenance Details for datafast-1.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datafast 1.0.1
File Interpreter ABI Platform Actions
datafast-1.0.1-py3-none-any.whl Python 3 none any Details for datafast-1.0.1-py3-none-any.whl

Total release size: 323.3 kB

Release files / datafast-1.0.1.tar.gz

Download URL datafast-1.0.1.tar.gz
Size 219.8 kB
Tags Source
SHA-256 checksum
How to use checksums
1933491ea58c6655de394ffafff3a8b79f1f8d234804d2f2e59df8a80ace58bf
BLAKE2b-256 checksum
How to use checksums
6f7b331d8f5bbca1d9e0e9c10598dcd9234ea507d6bdd15e0a43e80858702480
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Release files / datafast-1.0.1-py3-none-any.whl

Download URL datafast-1.0.1-py3-none-any.whl
Size 103.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
337775e1a893bf7cf9b1e3cbd31c7278b4507c885a68641a7f0085204f686e7e
BLAKE2b-256 checksum
How to use checksums
a1629b6bce22cda4297709f7d336db3b937fff42b17f6d074b188a8c4f4b09f3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.16

Security for datafast 1.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page