DataForge

Turn any website into a fine-tuning dataset — in one command.

Fine-tuning needs data, and good domain data is trapped in documentation sites, knowledge bases and public archives. Getting it out usually means writing a throwaway scraper, a chunker, a prompt loop and an exporter — then doing it all again for the next domain.

DataForge is that pipeline, already built, tuned, and checkpointed end to end:

sitemap ──▶ crawl ──▶ clean + chunk ──▶ LLM generates Q&A ──▶ score ──▶ JSONL / Parquet / HF Hub

pip install llm-web-crawler
dataforge init-recipe fema.yaml   # write a starter config
dataforge run fema.yaml           # crawl, generate, score, export — unattended

You get a versioned, deduplicated, quality-scored dataset in ChatML JSONL, Parquet and CSV, ready for Unsloth, Axolotl, TRL or HuggingFace datasets.

What you'll need

Before you run anything, here's the honest list of what's required vs. optional:

In short: one LLM (hosted key, or Ollama for zero-key/fully-local) is the only hard requirement. Everything else — which provider, which export target, whether you need a HuggingFace or Kaggle account — is a choice you make in the recipe, not a prerequisite to get started. Full setup for each option is in docs/INSTALLATION.md and docs/CONFIGURATION.md.

Why DataForge

A worked example

Building a U.S. disaster-preparedness dataset from Ready.gov. The full recipe ships with the project at examples/ready-gov.yaml:

version: 1
name: ready-gov-preparedness
stream: true

source:
  urls:
    - https://www.ready.gov/sitemap.xml
  language: en
  exclude: ['/node/', '/press-release', 're:toolkit']
  max_urls: 150

generation:
  format: qa
  goal: >
    Accurate Q&A about U.S. disaster preparedness, grounded strictly in
    official Ready.gov and FEMA guidance.
  n_per_chunk: 3

quality:
  threshold: 0.5

export:
  targets: [local]
  split:
    train: 0.8
    validation: 0.1
    test: 0.1
    group_by: page

dataforge run examples/ready-gov.yaml --dry-run   # validate + preview the plan
dataforge run examples/ready-gov.yaml             # execute

One sitemap becomes a few hundred curated pages, a few thousand chunks, and a scored, deduplicated, leak-free Q&A dataset — without answering a single prompt.

Get started

pip install llm-web-crawler
dataforge init-recipe my.yaml       # write an annotated starter recipe
dataforge run my.yaml --dry-run     # validate and print the plan, run nothing
dataforge run my.yaml               # execute end-to-end, no prompts

Prefer to explore a site interactively first? dataforge launches the guided wizard with a per-stage review. Full install options (uv, pip, standalone binaries, local models via Ollama), every CLI command, all environment variables and the complete recipe schema are in the docs:

• Installation — uv / pip / source / standalone binaries / Ollama
• Configuration — environment variables and full recipe reference
• Architecture — how discovery, streaming, quality and export fit together
• Development — project layout, testing, releasing
• Ethics & data residency — responsible use, PII, copyright, rate limits
• Third-party libraries — full dependency and license list
• Technical writeup — design rationale and references

What makes it different

Most scrapers stop at "here's the text." DataForge is built for the part that actually determines whether a fine-tune works:

• It knows a scraped dataset isn't i.i.d. Several Q&A pairs come from one chunk, several chunks from one page — so a naive random split leaks paraphrases across train/test and quietly inflates your eval score. DataForge splits by page and verifies the property before writing.
• It treats web failure modes as data, not noise. 429 isn't 404, Retry-After isn't ignored, and robots.txt Crawl-delay is honoured automatically — so a run against a real, imperfect site finishes instead of half-failing silently.
• It doesn't make you choose between fast and resumable. Streaming mode overlaps crawling and generation for throughput; every stage still checkpoints to SQLite, so a killed process loses only the item in flight.
• The quality bar is layered, not a single score. Length heuristics, a source-reference filter, deduplication, and an LLM judge that fails closed — with a rejection breakdown so you can see what's actually being filtered out.

License

MIT — see LICENSE for details.

If you use DataForge-generated datasets in a publication or project, attribution is appreciated but not required:

Ian Too. DataForge (2026). https://github.com/ianktoo/data-forge

@software{dataforge2026,
  author  = {Ian Too},
  title   = {DataForge: LLM Data Pipeline},
  year    = {2026},
  url     = {https://github.com/ianktoo/data-forge},
  license = {MIT}
}