DataForge

Turn any website into a fine-tuning dataset — in one command.

Current release: 2.6.0. An easier interactive menu: tasks in plain words, one screen per step, help at every stage, and "export what I have so far" at any point (discovered URLs, pages, chunks or samples). A missing or invalid API key now stops the run with one clear message instead of retrying every chunk. dataforge scrape saves pages and their tables with no AI, API key or cost. Tested on Linux, Windows and macOS with Python 3.11 to 3.14, with standalone binaries for Windows, Linux and Apple Silicon Macs. See the changelog.

Fine-tuning needs data, and good domain data is trapped in documentation sites, knowledge bases and public archives. Getting it out usually means writing a throwaway scraper, a chunker, a prompt loop and an exporter — then doing it all again for the next domain.

DataForge is that pipeline, already built, tuned, and checkpointed end to end:

sitemap ──▶ crawl ──▶ clean + chunk ──▶ LLM generates Q&A ──▶ score ──▶ JSONL / Parquet / HF Hub

pip install llm-web-crawler
dataforge init-recipe fema.yaml   # write a starter config
dataforge run fema.yaml           # crawl, generate, score, export — unattended

You get a versioned, deduplicated, quality-scored dataset in ChatML JSONL, Parquet and CSV, ready for Unsloth, Axolotl, TRL or HuggingFace datasets.

Discovery → Collection → Processing → Generation → Quality → Export

See docs/ARCHITECTURE.md for what each stage does.

What you'll need

Before you run anything, here's the honest list of what's required vs. optional:

In short: one LLM (hosted key, or a local model through Ollama or any OpenAI-compatible server) is the only hard requirement. Everything else — which provider, which export target, whether you need a HuggingFace or Kaggle account — is a choice you make in the recipe, not a prerequisite to get started. Full setup for each option is in docs/INSTALLATION.md and docs/CONFIGURATION.md.

Why DataForge

A worked example

Building a U.S. disaster-preparedness dataset from Ready.gov. The full recipe ships with the project at examples/ready-gov.yaml:

version: 1
name: ready-gov-preparedness
stream: true

source:
  urls:
    - https://www.ready.gov/sitemap.xml
  language: en
  exclude: ['/node/', '/press-release', 're:toolkit']
  max_urls: 150

generation:
  format: qa
  goal: >
    Accurate Q&A about U.S. disaster preparedness, grounded strictly in
    official Ready.gov and FEMA guidance.
  n_per_chunk: 3

quality:
  threshold: 0.5

export:
  targets: [local]
  split:
    train: 0.8
    validation: 0.1
    test: 0.1
    group_by: page

dataforge run examples/ready-gov.yaml --dry-run   # validate + preview the plan
dataforge run examples/ready-gov.yaml             # execute

One sitemap becomes a few hundred curated pages, a few thousand chunks, and a scored, deduplicated, leak-free Q&A dataset — without answering a single prompt.

Get started

pip install llm-web-crawler
dataforge init-recipe my.yaml       # write an annotated starter recipe
dataforge run my.yaml --dry-run     # validate and print the plan, run nothing
dataforge run my.yaml               # execute end-to-end, no prompts

Just need the pages, or a table on them? dataforge scrape fetches pages and saves their text as Markdown and every table as CSV and JSON. It needs no AI, no API key and spends nothing, and it obeys robots.txt and rate limits like everything else:

dataforge scrape https://www.husd.us/schools/middle-schools -o schools
# schools/page_001_table_1.csv:
# Middle Schools,Address,Phone,Fax
# Anthony Ochoa,"2121 Depot Road Hayward, CA 94545-2428",(510) 723-3130,(510) 786-0559
# ...

Add --check for rule-based checks (empty or very short pages, duplicate text), -f csv to write only the tables, and --json before the command for a machine-readable summary. AI agents get the same through the MCP tool scrape_page.

To update or remove DataForge, run dataforge update or dataforge uninstall yourself, in your own terminal. Both close DataForge first, so nothing is replaced while in use; uninstall asks whether to keep your data. Read the release notes before updating: a new version can change recipes, outputs or commands. See Updating and uninstalling.

Prefer to explore a site interactively first? dataforge launches the guided wizard with a per-stage review. Full install options (uv, pip, standalone binaries, local models via Ollama), every CLI command, all environment variables and the complete recipe schema are in the docs:

• Installation — uv / pip / source / standalone binaries / Ollama, plus known issues and workarounds
• Configuration — environment variables and full recipe reference
• Architecture — how discovery, streaming, quality and export fit together
• Development — project layout, testing, releasing
• Ethics & data residency — responsible use, PII, copyright, rate limits
• Third-party libraries — full dependency and license list
• Technical writeup — design rationale and references

Using DataForge from an AI agent

Claude Code, Codex, Cursor and other coding agents can drive DataForge through its non-interactive commands. Point the agent at the built-in guide:

dataforge agent-guide

It lists which commands are safe to run without a terminal, how to configure a provider through environment variables, the recipe workflow and exit codes, and the responsible-use rules an agent must follow before crawling a site.

For clients that support the Model Context Protocol, DataForge also runs as a local MCP server. Nothing is hosted; the client starts it on your machine:

pip install "llm-web-crawler[mcp]"
claude mcp add dataforge -- dataforge mcp   # run from your project directory

Claude Code picks up a newly added server only in a new session: exit with /exit, then run claude --continue in your terminal to reopen the same conversation with the DataForge tools loaded.

The agent then gets typed tools (scrape a page's text and tables, explore a site, validate a recipe, start and monitor a run, read session stats and samples) instead of parsing terminal output. Runs started this way must have a spending cap, and a recipe that disables robots.txt is refused.

Agents may never update, upgrade or uninstall DataForge. dataforge update and dataforge uninstall refuse to run without a terminal or inside an agent session, and the agent guide tells agents to leave it to you, after you have read the release notes.

What makes it different

Most scrapers stop at "here's the text." DataForge is built for the part that actually determines whether a fine-tune works:

• It knows a scraped dataset isn't i.i.d. Several Q&A pairs come from one chunk, several chunks from one page — so a naive random split leaks paraphrases across train/test and quietly inflates your eval score. DataForge splits by page and verifies the property before writing.
• It treats web failure modes as data, not noise. 429 isn't 404, Retry-After isn't ignored, and robots.txt Crawl-delay is honoured automatically — so a run against a real, imperfect site finishes instead of half-failing silently.
• It doesn't make you choose between fast and resumable. Streaming mode overlaps crawling and generation for throughput; every stage still checkpoints to SQLite, so a killed process loses only the item in flight.
• The quality bar is layered, not a single score. Length heuristics, a source-reference filter, deduplication, and an LLM judge that fails closed — with a rejection breakdown so you can see what's actually being filtered out.

License

MIT — see LICENSE for details.

If you use DataForge-generated datasets in a publication or project, attribution is appreciated but not required:

Too, I. K. (2026). DataForge: A Streaming, Leak-Aware Pipeline for Synthetic
LLM Fine-Tuning Datasets (Technical Note). Zenodo.
https://doi.org/10.5281/zenodo.22906071

@techreport{too2026dataforge,
  author      = {Too, Ian K.},
  title       = {{DataForge}: A Streaming, Leak-Aware Pipeline for Synthetic
                 {LLM} Fine-Tuning Datasets},
  type        = {Technical Note},
  institution = {Zenodo},
  year        = {2026},
  doi         = {10.5281/zenodo.22906071},
  url         = {https://doi.org/10.5281/zenodo.22906071}
}

GitHub's "Cite this repository" button reads the same details from CITATION.cff.

That DOI covers every version of the technical note and always resolves to the latest. To cite one exact version, use its own DOI: v1.1.0 is 10.5281/zenodo.22908858 and v1.0.0 is 10.5281/zenodo.22906072.