Prompt. Generate Synthetic Data. Train & Align Models.
DataDreamer is a powerful open-source Python library for prompting, synthetic data generation, and training workflows. It is designed to be simple, extremely efficient, and research-grade.
|
Installation pip3 install datadreamer.dev
|
|
demo.py |
Result of demo.py |
|---|---|
See the full demo script |
See the synthetic dataset and the trained model |
|
🚀 For more demonstrations and recipes see the Quick Tour page. |
|
With DataDreamer you can:
- 💬 Create Prompting Workflows: Create and run multi-step, complex, prompting workflows easily with major open source or API-based LLMs.
- 📊 Generate Synthetic Datasets: Generate synthetic datasets for novel tasks or augment existing datasets with LLMs.
- ⚙️ Train Models: Align models. Fine-tune models. Instruction-tune models. Distill models. Train on existing data or synthetic data.
- ... learn more about what's possible in the Overview Guide
DataDreamer is:
- 🧩 Simple: Simple and approachable to use with sensible defaults, yet powerful with support for bleeding edge techniques.
- 🔬 Research-Grade: Built for researchers, by researchers, but accessible to all. A focus on correctness, best practices, and reproducibility.
- 🏎️ Efficient: Aggressive caching and resumability built-in. Support for techniques like quantization, parameter-efficient training (LoRA), and more.
- 🔄 Reproducible: Workflows built with DataDreamer are easily shareable, reproducible, and extendable.
- 🤝 Makes Sharing Easy: Publishing datasets and models is simple. Automatically generate data cards and model cards with metadata. Generate a list of any citations required.
- ... learn more about the motivation and design principles behind DataDreamer.
Citation
coming soon...
Contact
Please reach out to us via email (ajayp@upenn.edu) or on Discord if you have any questions, comments, or feedback.
Copyright © 2024, Ajay Patel. Released under the MIT License.
Thank you to the maintainers at Hugging Face and LiteLLM for accepting contributions neccessary for DataDreamer and providing upstream support.
1 maintainer
Metadata
1 maintainer
Release files for datadreamer.dev 0.11.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| datadreamer_dev-0.11.0.tar.gz | 274.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| datadreamer_dev-0.11.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 641.3 kB
Release files / datadreamer_dev-0.11.0.tar.gz
| Download URL | datadreamer_dev-0.11.0.tar.gz |
|---|---|
| Size | 274.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0230dc1503945d945d822b0245cc4982939c68e864ca979615de3497eb36b7c7
|
|
BLAKE2b-256 checksum How to use checksums |
7023dd5785762c50ade3a243ce27bc8d3c3de1ac7b10bf9a18768781bdd0ffb2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.7.1 CPython/3.10.9 Linux/6.2.0-1017-aws
|
Release files / datadreamer_dev-0.11.0-py3-none-any.whl
| Download URL | datadreamer_dev-0.11.0-py3-none-any.whl |
|---|---|
| Size | 366.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f7010d26c863fef67e1ea8339bb797ebb1119979a80a4a8934318b7225c70c3b
|
|
BLAKE2b-256 checksum How to use checksums |
490e2def4e5ed4ed3bf351682c13e4bfdb817da427eeef98796cd8432571f212
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.7.1 CPython/3.10.9 Linux/6.2.0-1017-aws
|
