Generate high-quality and diverse synthetic data with LLMs.
Documentation · Quickstart · Cookbook · Examples · API reference
Datafast is a Python library for generating synthetic datasets with LLMs.
You assemble building blocks for data operations into a custom pipeline that generates a synthetic dataset matching your project's needs. It works with open-weight and proprietary models.
Start with seeds or the dimensions your dataset must cover. Connect LLMs and data processing steps for filtering, transformations, and annotations, then run the pipeline. Save the dataset locally or push it to the Hugging Face Hub.
Install
pip install datafast
Datafast needs Python 3.11 or newer. Extras add Parquet, the Hugging Face Hub and Langfuse tracing: see Installation.
Quick example
This example generates exam questions and answers for 3 science topics at 2 levels, beginner and advanced. The result is 6 example records.
Set the key for your provider, here OpenAI:
export OPENAI_API_KEY="sk-..."
from datafast import LLMStep, Seed, Sink, openai
pipeline = (
Seed.product(
Seed.values("topic", ["photosynthesis", "plate tectonics", "vaccines"]),
Seed.values("level", ["beginner", "advanced"]),
)
>> LLMStep(
prompt=(
"Write one {level} exam question about {topic}, with its answer. "
"Return JSON with fields question and answer."
),
input_columns=["topic", "level"],
output_columns=["question", "answer"],
parse_mode="json",
model=openai(),
)
>> Sink.jsonl("questions.jsonl")
)
pipeline.run()
The pipeline has 3 steps:
Seed.productmakes 1 record for each combination of the values in its dimensions: 3 topics × 2 levels = 6 records.LLMStepmakes 1 LLM call for each record. Withparse_mode="json", it splits the response into thequestionandanswercolumns.Sink.jsonlwrites the records toquestions.jsonl.
Each line of the file is 1 record. The seed columns stay beside the generated columns,
and _model tells you which model wrote the record:
{
"topic": "photosynthesis",
"level": "beginner",
"question": "What process do plants use to convert light into chemical energy?",
"answer": "Photosynthesis, which occurs in the chloroplasts.",
"_model": "gpt-6-luna"
}
The Quickstart explains each step in more detail. Key concepts and the API reference list all the steps.
Use cases
- Make evaluation data for an LLM project before you have real user data.
- Add records to an existing dataset to cover more cases.
- Test an LLM application on realistic inputs before you ship it.
- Make datasets to fine-tune LLMs and agents.
- Distill a large model into a smaller one: the large model writes the training data.
Why datafast
- You declare the coverage. You give the values of each dimension, and
Seed.productmakes all the combinations. You do not write nested loops. - A long run can resume after a crash. Set
checkpoint_dirto save progress, then restart with the same directory andresume=True. See Checkpoints and resume for recovery limits. - You can change the provider in 1 line. The same provider factories work for OpenAI, Anthropic, Gemini, Mistral, OpenRouter, Ollama and all OpenAI-compatible servers such as vLLM and llama.cpp. See Served models.
- Datafast finds mistakes before you pay.
Pipeline.compile()checks the step order and all column references before the first LLM call. - Each record shows its origin. The seed values and the model id stay in the record, so you can filter or audit the dataset later.
Cookbook
Each cookbook is a complete pipeline that makes a real dataset. The script is in
examples/scripts/, and the page explains each step.
| Cookbook | Dataset | Script |
|---|---|---|
| Text classification | Trail reports from hikers in 2 languages, with labels | 45 |
| Personas | Personas made from real articles | 43 |
| Domain text corpus | Spacecraft engineering text in 2 languages | 44 |
| Preference pairs | Chosen and rejected answers with scores, for RLHF | 42 |
The examples page lists all 45 scripts. Scripts 01 to 14 do not use an API key.
Contributing
Bug reports, ideas and pull requests are welcome. Open an issue or read the contributing guide.
License
Datafast is released under the Apache 2.0 license.
If datafast helps you, give the repo a star. It helps other people find it.
Metadata
Release files for datafast 1.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| datafast-1.0.1.tar.gz | 219.8 kB | Without provenance | Details for datafast-1.0.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| datafast-1.0.1-py3-none-any.whl | Python 3 | none | any | Details for datafast-1.0.1-py3-none-any.whl |
Total release size: 323.3 kB
Release files / datafast-1.0.1.tar.gz
| Download URL | datafast-1.0.1.tar.gz |
|---|---|
| Size | 219.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1933491ea58c6655de394ffafff3a8b79f1f8d234804d2f2e59df8a80ace58bf
|
|
BLAKE2b-256 checksum How to use checksums |
6f7b331d8f5bbca1d9e0e9c10598dcd9234ea507d6bdd15e0a43e80858702480
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.16
|
Release files / datafast-1.0.1-py3-none-any.whl
| Download URL | datafast-1.0.1-py3-none-any.whl |
|---|---|
| Size | 103.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
337775e1a893bf7cf9b1e3cbd31c7278b4507c885a68641a7f0085204f686e7e
|
|
BLAKE2b-256 checksum How to use checksums |
a1629b6bce22cda4297709f7d336db3b937fff42b17f6d074b188a8c4f4b09f3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.16
|
Security for datafast 1.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.