Prompt. Generate Synthetic Data. Train & Align Models.
DataDreamer is a powerful open-source Python library for prompting, synthetic data generation, and training workflows. It is designed to be simple, extremely efficient, and research-grade.
|
Installation pip3 install datadreamer.dev
|
|
demo.py |
Result of demo.py |
|---|---|
See the full demo script |
See the synthetic dataset and the trained model |
|
🚀 For more demonstrations and recipes see the Quick Tour page. |
|
With DataDreamer you can:
- 💬 Create Prompting Workflows: Create and run multi-step, complex, prompting workflows easily with major open source or API-based LLMs.
- 📊 Generate Synthetic Datasets: Generate synthetic datasets for novel tasks or augment existing datasets with LLMs.
- ⚙️ Train Models: Align models. Fine-tune models. Instruction-tune models. Distill models. Train on existing data or synthetic data.
- ... learn more about what's possible in the Overview Guide
DataDreamer is:
- 🧩 Simple: Simple and approachable to use with sensible defaults, yet powerful with support for bleeding edge techniques.
- 🔬 Research-Grade: Built for researchers, by researchers, but accessible to all. A focus on correctness, best practices, and reproducibility.
- 🏎️ Efficient: Aggressive caching and resumability built-in. Support for techniques like quantization, parameter-efficient training (LoRA), and more.
- 🔄 Reproducible: Workflows built with DataDreamer are easily shareable, reproducible, and extendable.
- 🤝 Makes Sharing Easy: Publishing datasets and models is simple. Automatically generate data cards and model cards with metadata. Generate a list of any citations required.
- ... learn more about the motivation and design principles behind DataDreamer.
Citation
coming soon...
Contact
Please reach out to us via email (ajayp@upenn.edu) or on Discord if you have any questions, comments, or feedback.
Copyright © 2024, Ajay Patel. Released under the MIT License.
Thank you to the maintainers at Hugging Face and LiteLLM for accepting contributions neccessary for DataDreamer and providing upstream support.
1 maintainer
Metadata
1 maintainer
Release files for datadreamer.dev 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| datadreamer_dev-0.3.0.tar.gz | 267.0 kB | Without provenance | Details for datadreamer_dev-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| datadreamer_dev-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for datadreamer_dev-0.3.0-py3-none-any.whl |
Total release size: 622.5 kB
Release files / datadreamer_dev-0.3.0.tar.gz
| Download URL | datadreamer_dev-0.3.0.tar.gz |
|---|---|
| Size | 267.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
531975ec0d28eb89dd3994a1ed44240196528125f8cdfd04eb7a6c5ade184358
|
|
BLAKE2b-256 checksum How to use checksums |
28c6bf1099346723a23054e7922fb1ca6347121144d3cbe6175f2b3444e57a4a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.7.1 CPython/3.10.9 Linux/6.2.0-1017-aws
|
Release files / datadreamer_dev-0.3.0-py3-none-any.whl
| Download URL | datadreamer_dev-0.3.0-py3-none-any.whl |
|---|---|
| Size | 355.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
79655828cf651e2b076fc1475b8ce57ad6f55f18a3b95ad7c6b43724bbcceef5
|
|
BLAKE2b-256 checksum How to use checksums |
cd8e2c417bfafb213460721b7fd74c0db90092cae33a727ab0d7a042589d22b7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.7.1 CPython/3.10.9 Linux/6.2.0-1017-aws
|
Security for datadreamer.dev 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
