docetl 0.3.0

ETL with LLM operations.

DocETL: Declarative & Agentic Map-Reduce

What is DocETL · Install · Python API · YAML · DocWrangler UI · Docs

---

What is DocETL

DocETL helps you process large collections of data (structured and unstructured) with LLMs. You write each operation in natural language, e.g., "pull out every complaint in this ticket," and DocETL

• provides the operators you need (map, reduce, filter, and more) and orchestrates them, parallelizing work across your data,
• optimizes your pipeline automatically, swapping models, rewriting prompts, decomposing operations, and replacing subtasks with code wherever possible, to raise accuracy and cut cost, and
• returns tables, easy to query in your favorite database.

Without DocETL, you write each LLM call yourself, wire them together, and tune the result for accuracy, cost, and latency by hand.

CLI	DocWrangler UI

---

Install

pip install docetl
export OPENAI_API_KEY=your_key   # or any LLM provider key

---

Need Help Writing Your Pipeline?

Use Claude Code (recommended): run docetl install-skill and describe your task. See the quickstart.

If you'd rather use ChatGPT or the Claude app, copy the prompt at docetl.org/llms-full.txt into the chat before describing your task.

---

Python API (recommended)

Best for production code, notebooks, and scripting. Full guide

import docetl

docetl.default_model = "gpt-4o-mini"
docetl.rate_limits = {
    "llm_call": [{"count": 500, "per": 1, "unit": "minute"}],
    "llm_tokens": [{"count": 200_000, "per": 1, "unit": "minute"}],
}

# Classify support tickets, then summarize each category
pipeline = docetl.read_json("tickets.json")

pipeline = pipeline.map(
    prompt="Classify this support ticket: {{ input.text }}",
    output={"schema": {"category": "str", "priority": "str"}},
)

pipeline = pipeline.reduce(
    reduce_key="category",
    prompt="Summarize these tickets: {% for t in inputs %}{{ t.text }}{% endfor %}",
    output={"schema": {"summary": "str"}},
)

pipeline.schema()  # {'category': 'str', 'summary': 'str'}
pipeline.show()  # run on 5 docs and print results
rows = pipeline.collect()  # full run
print(f"Cost: ${pipeline.total_cost:.4f}")

---

YAML (low-code)

Declare your pipeline in a config file, no Python needed. Tutorial

datasets:
  tickets:
    type: file
    path: tickets.json

default_model: gpt-4o-mini

operations:
  - name: classify
    type: map
    prompt: "Classify this support ticket and assign a priority level."
    output:
      schema:
        category: str
        priority: str

pipeline:
  steps:
    - name: triage
      input: tickets
      operations: [classify]
  output:
    type: file
    path: output.json

docetl run pipeline.yaml

---

DocWrangler UI

Visual playground for interactive prompt development. Edit prompts, see results in real time. Try it at docetl.org/playground or run it locally.

---

Documentation

Python API Guide	Frame API reference: operations, config, optimization
YAML Tutorial	Step-by-step walkthrough of declarative pipelines
Operators	Map, filter, reduce, resolve, split, gather, extract, and more
Optimization	Automatic cost-accuracy optimization with MOAR
DocWrangler Setup	Run the interactive UI locally or via Docker
Claude Code Quick Start	Describe your task and let Claude build the pipeline

---

Community

Discord · Conversation Generator · Text-to-Speech · YouTube Transcript Topics

---

Development

git clone https://github.com/ucbepic/docetl.git && cd docetl
make install
make tests-basic  # < $0.01 with OpenAI

---

Papers

DocETL was created at the EPIC Data Lab and Data Systems and Foundations group at UC Berkeley.

DocETL, VLDB 2025 (paper)

@article{shankar2025docetl,
  title={DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing},
  author={Shankar, Shreya and Chambers, Tristan and Shah, Tarak and Parameswaran, Aditya G and Wu, Eugene},
  journal={Proceedings of the VLDB Endowment},
  volume={18}, number={9}, pages={3035--3048}, year={2025}
}

DocWrangler, UIST 2025, Best Paper Honorable Mention (paper)

@inproceedings{shankar2025docwrangler,
  title={Steering Semantic Data Processing With DocWrangler},
  author={Shankar*, Shreya and Chopra*, Bhavya and Hasan, Mawil and Lee, Stephen and Hartmann, Bj{\"o}rn and Hellerstein, Joseph M and Parameswaran, Aditya G and Wu, Eugene},
  booktitle={Proceedings of the ACM Symposium on User Interface Software and Technology (UIST)},
  year={2025}
}

MOAR, VLDB 2026 (paper)

@article{wei2026moar,
  title={Multi-Objective Agentic Rewrites for Unstructured Data Processing},
  author={Wei*, Lindsey Linxi and Shankar*, Shreya and Zeighami, Sepanta and Chung, Yeounoh and Ozcan, Fatma and Parameswaran, Aditya G},
  journal={Proceedings of the VLDB Endowment}, year={2026}
}

*Co-first authors