DataForge

Turn any website into a fine-tuning dataset — in one command.

Current release: 2.4.1. AI agents can drive DataForge through a local MCP server or the built-in dataforge agent-guide, robots.txt Crawl-delay is enforced in every stage, and the MCP inspection tools (session_stats, view_samples) now work after a real run. See the changelog.

Fine-tuning needs data, and good domain data is trapped in documentation sites, knowledge bases and public archives. Getting it out usually means writing a throwaway scraper, a chunker, a prompt loop and an exporter — then doing it all again for the next domain.

DataForge is that pipeline, already built, tuned, and checkpointed end to end:

sitemap ──▶ crawl ──▶ clean + chunk ──▶ LLM generates Q&A ──▶ score ──▶ JSONL / Parquet / HF Hub

pip install llm-web-crawler
dataforge init-recipe fema.yaml   # write a starter config
dataforge run fema.yaml           # crawl, generate, score, export — unattended

You get a versioned, deduplicated, quality-scored dataset in ChatML JSONL, Parquet and CSV, ready for Unsloth, Axolotl, TRL or HuggingFace datasets.

Discovery → Collection → Processing → Generation → Quality → Export

See docs/ARCHITECTURE.md for what each stage does.

What you'll need

Before you run anything, here's the honest list of what's required vs. optional:

In short: one LLM (hosted key, or Ollama for zero-key/fully-local) is the only hard requirement. Everything else — which provider, which export target, whether you need a HuggingFace or Kaggle account — is a choice you make in the recipe, not a prerequisite to get started. Full setup for each option is in docs/INSTALLATION.md and docs/CONFIGURATION.md.

Why DataForge

A worked example

Building a U.S. disaster-preparedness dataset from Ready.gov. The full recipe ships with the project at examples/ready-gov.yaml:

version: 1
name: ready-gov-preparedness
stream: true

source:
  urls:
    - https://www.ready.gov/sitemap.xml
  language: en
  exclude: ['/node/', '/press-release', 're:toolkit']
  max_urls: 150

generation:
  format: qa
  goal: >
    Accurate Q&A about U.S. disaster preparedness, grounded strictly in
    official Ready.gov and FEMA guidance.
  n_per_chunk: 3

quality:
  threshold: 0.5

export:
  targets: [local]
  split:
    train: 0.8
    validation: 0.1
    test: 0.1
    group_by: page

dataforge run examples/ready-gov.yaml --dry-run   # validate + preview the plan
dataforge run examples/ready-gov.yaml             # execute

One sitemap becomes a few hundred curated pages, a few thousand chunks, and a scored, deduplicated, leak-free Q&A dataset — without answering a single prompt.

Get started

pip install llm-web-crawler
dataforge init-recipe my.yaml       # write an annotated starter recipe
dataforge run my.yaml --dry-run     # validate and print the plan, run nothing
dataforge run my.yaml               # execute end-to-end, no prompts

Prefer to explore a site interactively first? dataforge launches the guided wizard with a per-stage review. Full install options (uv, pip, standalone binaries, local models via Ollama), every CLI command, all environment variables and the complete recipe schema are in the docs:

• Installation — uv / pip / source / standalone binaries / Ollama
• Configuration — environment variables and full recipe reference
• Architecture — how discovery, streaming, quality and export fit together
• Development — project layout, testing, releasing
• Ethics & data residency — responsible use, PII, copyright, rate limits
• Third-party libraries — full dependency and license list
• Technical writeup — design rationale and references

Using DataForge from an AI agent

Claude Code, Codex, Cursor and other coding agents can drive DataForge through its non-interactive commands. Point the agent at the built-in guide:

dataforge agent-guide

It lists which commands are safe to run without a terminal, how to configure a provider through environment variables, the recipe workflow and exit codes, and the responsible-use rules an agent must follow before crawling a site.

For clients that support the Model Context Protocol, DataForge also runs as a local MCP server. Nothing is hosted; the client starts it on your machine:

pip install "llm-web-crawler[mcp]"
claude mcp add dataforge -- dataforge mcp   # run from your project directory

Claude Code picks up a newly added server only in a new session: exit with /exit, then run claude --continue in your terminal to reopen the same conversation with the DataForge tools loaded.

The agent then gets typed tools (explore a site, validate a recipe, start and monitor a run, read session stats and samples) instead of parsing terminal output. Runs started this way must have a spending cap, and a recipe that disables robots.txt is refused.

What makes it different

Most scrapers stop at "here's the text." DataForge is built for the part that actually determines whether a fine-tune works:

• It knows a scraped dataset isn't i.i.d. Several Q&A pairs come from one chunk, several chunks from one page — so a naive random split leaks paraphrases across train/test and quietly inflates your eval score. DataForge splits by page and verifies the property before writing.
• It treats web failure modes as data, not noise. 429 isn't 404, Retry-After isn't ignored, and robots.txt Crawl-delay is honoured automatically — so a run against a real, imperfect site finishes instead of half-failing silently.
• It doesn't make you choose between fast and resumable. Streaming mode overlaps crawling and generation for throughput; every stage still checkpoints to SQLite, so a killed process loses only the item in flight.
• The quality bar is layered, not a single score. Length heuristics, a source-reference filter, deduplication, and an LLM judge that fails closed — with a rejection breakdown so you can see what's actually being filtered out.

License

MIT — see LICENSE for details.

If you use DataForge-generated datasets in a publication or project, attribution is appreciated but not required:

Ian Too. DataForge (2026). https://github.com/ianktoo/data-forge

@software{dataforge2026,
  author  = {Ian Too},
  title   = {DataForge: LLM Data Pipeline},
  year    = {2026},
  url     = {https://github.com/ianktoo/data-forge},
  license = {MIT}
}