Brainbrew

The ridiculously easy, stupidly powerful no-code machine that turns your boring PDFs and TXT files into god-tier synthetic LLM training data

Brainbrew — Think of it like a mad scientist + coffee machine combo: you dump in documents, hit one button, and BOOM — fresh, high-quality instruction datasets appear like magic. No coding. No spreadsheets. No crying over JSON formatting at 3 a.m.

We took the original prototype, slayed every bug, rebuilt generation so every answer is grounded in your documents and checked by an LLM judge, added semantic chunking, multi-model ensemble, near-duplicate and paraphrase removal, benchmark decontamination, PII scrubbing, quality scoring, progress bars, Docker, and a bunch of other goodies… then wrapped it in a shiny Streamlit UI that even your grandma could use.

Current version: v3.0.2

Drop in any knowledge — PDFs, text, books, docs — and let Brainbrew do the rest.

Docs: yog-sotho.github.io/Brainbrew, built from the Markdown in docs/; preview your edits locally with uv run --group docs mkdocs serve.

Why Brainbrew Slaps

• Zero coding — literally just upload files and click "Generate Dataset"
• Grounded Q&A — the model writes several kinds of questions per passage (factual, conceptual, procedural, comparative, multi-hop) and answers each one from that passage only
• LLM judge — Balanced and Research mode score every pair for faithfulness, helpfulness and correctness and keep only the good ones
• Real target size — Brainbrew keeps generating until it reaches the number you asked for (or your documents run out), and tells you up front what your documents can support
• Multi-model ensemble — comma-separate your models for diverse, high-quality output
• Semantic chunking — paragraph-aware document splitting that respects topic boundaries
• Dataset deduplication — near-duplicates via MinHash-LSH, plus optional paraphrase removal with an embedding model
• Benchmark decontamination — optionally drop pairs that overlap GSM8K, MMLU, ARC, TruthfulQA or HumanEval
• Quality scoring — SUPER / GOOD / NORMAL / BAD / DISASTER grades after generation
• 4 export formats — Alpaca, ShareGPT, ChatML, and OpenAI fine-tuning JSONL
• Any OpenAI-compatible endpoint — OpenAI, a local vllm serve, Ollama, llama.cpp or your own URL; fully async with retries and parallel requests
• Auto LoRA training — optional one-click fine-tune with TRL + PEFT (QLoRA on NVIDIA GPUs)
• Hugging Face publish — one checkbox and your dataset is on the Hub (private by default) with an auto-generated dataset card; the LoRA adapter can go up too, with a model card
• Runs in the background — close or reload the tab and the run keeps going; follow it live, cancel it, or find it later on the Run history page
• Every run is kept — dataset, adapter, rejected pairs, a log and a manifest (models, seed, token usage, actual cost, stage timings, every filter count) land in runs/<run-id>/
• Headless CLI — brainbrew run docs/*.pdf --config cfg.yaml for scripts and servers, same pipeline as the UI
• Error handling & progress bars — failures are shown in the UI and recorded in the run manifest
• Docker ready — run it anywhere without summoning the dependency demon
• ~700 automated tests — the real pipeline against an in-memory model server, and real LoRA training, run in CI on every PR; a nightly benchmark checks generation quality with a real model

In short: it's what every AI guy wanted and never found anywhere.

Features

• Quality Modes: Fast (filters only, cheap & quick), Balanced (+ LLM judge, the sweet spot), Research (+ harder evolved questions, still checked against the source)
• Output Formats: Alpaca, ShareGPT, ChatML, OpenAI — pick what your training framework needs
• Clean & sanitize: PII redaction (emails, phones, Luhn-checked cards, IBANs, IPs, links; optional Presidio for names), HTML cleanup and quality gates, applied the same way to every export format
• Refusal filter: "As an AI…" answers, non-answers and prompt leaks never reach your dataset
• Quality dashboard: grade, record count, answer length and uniqueness after every run
• Multi-Model Ensemble: Split prompts across multiple teacher models for diversity
• Deduplication: MinHash near-duplicates (fast enough for tens of thousands of pairs) + optional embedding-based paraphrase removal
• Cost & yield estimator: See estimated cost, and how many pairs your documents can support, before you click Generate
• Live Stats: Record count, average output length, uniqueness ratio
• Dataset Preview: See the first 5 examples before downloading
• Pydantic Config: Type-safe everything (no more surprise crashes)

Quick Start (Takes 2 Minutes)

1. Clone & Setup

git clone https://github.com/Yog-Sotho/Brainbrew.git
cd Brainbrew

2. Run the installer (Python 3.12 or 3.13)

bash install.sh

The installer handles everything: Python version check, virtual environment, pip dependencies, GPU detection, and .env setup.

3. Or install manually

Dependencies are locked in uv.lock. Add the extras you need:

# with uv (recommended)
uv sync                    # core: any OpenAI-compatible endpoint, any OS
uv sync --extra train      # + LoRA training with TRL + PEFT (an NVIDIA GPU for real models)
uv sync --extra pii        # + Presidio name detection (then: python -m spacy download en_core_web_lg)
uv tool install --python 3.12 vllm==0.31.0   # optional: `vllm serve`, a local GPU model server (Linux, NVIDIA), in its own environment
or with pip (hash-pinned exports of the same lock)
python3.12 -m venv .venv && source .venv/bin/activate
pip install --require-hashes -r requirements.txt          # or requirements-train.txt (core + LoRA training)
cp .env.sample .env

Edit .env:

OPENAI_API_KEY=sk-...
HF_TOKEN=hf_...
HF_USERNAME=yourusername

4. Run It

uv run streamlit run app.py

Or without a browser:

uv run brainbrew run notes.pdf --model gpt-4o-mini --size 200 --format sharegpt -o dataset.jsonl
uv run brainbrew run docs/*.pdf --config cfg.yaml   # any setting, YAML or JSON; secrets stay in the environment
uv run brainbrew runs list                          # recent runs (web and CLI share the runs folder)
uv run brainbrew runs show <run-id>                # the full manifest

Installed with install.sh or pip instead? Activate the environment (source .venv/bin/activate), then run streamlit run app.py and python cli.py run …: those routes install the dependencies, not the brainbrew command.

Ctrl-C cancels a CLI run cleanly (it is recorded as cancelled). Exit codes: 0 success, 1 failed, 2 bad input, 130 cancelled.

No API key? Run a model on your own GPU and pick Local vLLM server in the sidebar:

vllm serve Qwen/Qwen3-4B-Instruct-2507      # or: bash run_vllm.sh (written by install.sh on GPUs with 24 GB+ VRAM)

vLLM is not a Brainbrew dependency: it is a separate server Brainbrew talks to over HTTP, so it lives in its own environment (uv tool install, install.sh's .venv-vllm, or the official image in compose.yaml) with its own pinned packages.

Boom. Browser opens. You're now a dataset wizard.

Docker (For the Cool Kids)

# Brainbrew + a local vLLM model server, no API key needed (NVIDIA GPU + Container Toolkit)
docker compose up -d                                   # model: VLLM_MODEL (default Qwen/Qwen3-4B-Instruct-2507)
CPU image: OpenAI or any OpenAI-compatible endpoint, ~1.2 GB. Pull the published one
(linux/amd64 + linux/arm64, from the next release on) or build it yourself:
docker pull ghcr.io/yog-sotho/brainbrew && docker tag ghcr.io/yog-sotho/brainbrew brainbrew-api
docker build --target api -t brainbrew-api .
docker run -p 127.0.0.1:8501:8501 --env-file .env brainbrew-api
Headless CLI in the CPU image
docker run --rm -v "$PWD:/data" -v brainbrew-runs:/app/runs -e OPENAI_API_KEY brainbrew-api 

python cli.py run /data/notes.pdf -o /data/dataset.jsonl
GPU image (LoRA training). Needs NVIDIA driver R580+ (CUDA 13.0). For a model server, use compose above.
docker build -t brainbrew .
docker run --gpus all -p 127.0.0.1:8501:8501 --env-file .env brainbrew
Keep runs (datasets, adapters) across container restarts
docker run --gpus all -p 127.0.0.1:8501:8501 --env-file .env -v brainbrew-runs:/app/runs brainbrew

Images run as a non-root user, and .env / .streamlit/secrets.toml are never copied into them (see .dockerignore). In compose.yaml the app can only talk to its vLLM container (BRAINBREW_ALLOW_CUSTOM_ENDPOINTS=0), and vLLM's port is not published on the host.

Or use the installer:

bash install.sh --docker

Open http://localhost:8501 and flex.

Security

• The app listens on 127.0.0.1 by default (.streamlit/config.toml); the Docker examples publish the port on localhost only.
• API keys from .env stay on the server and are never sent to the browser. The sidebar only says a server key is configured.
• The server's OPENAI_API_KEY is only ever sent to the server's own endpoint (OPENAI_BASE_URL, or OpenAI). If a visitor picks another endpoint, only a key they type themselves is used. Set BRAINBREW_ALLOW_CUSTOM_ENDPOINTS=0 to lock the app to the server endpoint. With login on, the endpoint is locked unless you set it to 1; with login on, the server's HF_TOKEN only publishes to repos under HF_USERNAME.
• With login on, the gate runs on every page, and each user only sees their own runs (a run id in a URL is not enough to open someone else's).
• Before exposing Brainbrew to a network, turn on login: set BRAINBREW_REQUIRE_LOGIN=1 and add an OIDC provider in .streamlit/secrets.toml (template: .streamlit/secrets.toml.example). Without that config the app refuses to start the UI.

Operator settings (environment)

How to Use (So Easy It's Embarrassing)

1. Upload your PDFs or TXT files (multiple OK!)
2. Pick a model endpoint (OpenAI, a local vLLM or Ollama server, or a custom URL) and your teacher model
3. Optionally enter multiple models comma-separated for ensemble diversity
4. Choose quality mode (Fast / Balanced / Research)
5. Choose output format (Alpaca / ShareGPT / ChatML / OpenAI)
6. Slide to the dataset size you want. Brainbrew shows how many pairs your documents can support (about 6 per 1,600-character chunk) and warns you if you ask for more
7. Optional: semantic chunking, deduplication, sanitizing, benchmark decontamination, LoRA training, HF publish
8. Smash the big Generate Dataset button. The run starts in the background: you can reload, close the tab or start another one in a second tab. The page URL (?run=…) brings you back to it
9. Check your quality score, preview examples, and download. Older runs are on the Run history page, with their details, rejected pairs and log

Each run gets its own folder, runs/<run-id>/ (or $BRAINBREW_RUNS_DIR): the source text, the generated and cleaned records, the exported dataset, the LoRA adapter (zipped) and a manifest.json with the settings (never your keys), generation and filter counts, token usage per model and the quality report.

Done. Go train a model that actually knows your niche.

High-quality Q&A pairs stream straight into your model's brain. Automated study, zero effort.

Advanced Settings (Sidebar)

• Model endpoint — OpenAI, a local vLLM server (localhost:8000), Ollama (localhost:11434) or a custom URL. Operators can set the default with OPENAI_BASE_URL and the default model with BRAINBREW_DEFAULT_MODEL; local servers need no key
• API Key — for OpenAI or any hosted OpenAI-compatible provider
• HF Token — for publishing
• Semantic Chunking — paragraph-aware splitting (experimental)
• Deduplication — remove near-duplicate question/answer pairs
• Clean & sanitize — PII redaction with a choice for links (keep the site only / remove / keep) and optional Presidio name detection
• Remove benchmark overlap — drop pairs sharing a 13-word passage with the selected public test sets
• Generation settings — temperature, max answer length, parallel requests, request timeout, judge model and minimum judge score, and an embedding model for paraphrase removal

In the main panel, under the Auto-train and Publish checkboxes:

• Publishing — license for the dataset card, Make it public (repos are private otherwise), and Also publish the LoRA adapter (as <dataset repo>-lora)
• LoRA settings — base model (default Qwen/Qwen3-4B-Instruct-2507) and rank, shown when auto-train is on

Tech Stack

• Streamlit – beautiful UI
• openai SDK (async) – one code path for OpenAI, vLLM, Ollama and friends, with structured outputs
• vLLM – GPU wizardry, as a separate vllm serve process
• MinHash-LSH (numpy + xxhash) – near-duplicate removal that scales
• TRL + PEFT – LoRA / QLoRA fine-tuning, loss on answers only
• LangChain text splitters – character & semantic chunking
• Pydantic + Structlog – no more "it worked on my machine" excuses
• pytest – ~700 tests with CI via GitHub Actions
• Typer + Rich – the brainbrew command line
• uv – locked, hash-verified dependencies

Hardware Requirements

Pro tip: Start with OpenAI mode. Once it works, flex with vLLM on RunPod/Modal.

Troubleshooting

• "CUDA out of memory" — Use a smaller model in vllm serve, or lower --gpu-memory-utilization
• Rate limits or timeouts — Lower Parallel requests or raise Request timeout in Generation settings (slow CPU servers need 1–2 parallel requests)
• Fewer pairs than you asked for — your documents ran out of material; the manifest's generation block shows how many were filtered, judged out or duplicates
• Nothing happens — Check console + make sure you uploaded files
• HF publish fails — Token wrong? Repo name taken? Classic.
• bitsandbytes error — Needs CUDA. Expected on CPU-only machines.
• A run failed — the error is shown in the app and saved as error in runs/<run-id>/manifest.json. Interrupted runs cannot be resumed; start a new one.

Still stuck? Open an issue. We'll roast the bug together.

Testing

Brainbrew ships with about 700 automated tests. Pipeline tests run the real async engine and openai SDK against an in-memory OpenAI-compatible server that can misbehave on demand (rate limits, broken JSON, refusals, missing structured-output support); app tests drive the real Streamlit app (upload → generate → download); LoRA tests run real TRL + PEFT training on tiny models. Also covered: config validation, security (API keys never reach logs, run folders, the browser or a URL the visitor chose), all four export formats, PII, dedup, decontamination and HF publishing. No GPU or API key required.

uv run pytest                          # all tests (uses the locked core env + dev tools)
uv run pytest tests/test_security.py   # just security tests
# LoRA contract tests need the training packages (CI installs CPU torch for them)
uv run ruff check . && uv run mypy

CI (.github/workflows/ci.yml) runs lint, type checks, tests with an 80% coverage gate, real LoRA training on CPU, lockfile consistency, pip-audit, gitleaks, and a Docker build + smoke test on every push and PR.

Quality benchmark. bench/run_bench.py generates datasets from three public-domain PDFs in tests/fixtures/bench/ and checks the generation gate: average judge faithfulness ≥ 4.0 (from an independent judge pass), near-duplicate rate < 2 %, no refusals, and yield within 10 % of the target. .github/workflows/bench.yml runs it nightly. It can use OpenAI (OPENAI_API_KEY), Google Gemini's free tier (set the GEMINI_API_KEY secret and the BENCH_PROVIDER=gemini variable), or any OpenAI-compatible endpoint; see docs/testing.md.

OPENAI_API_KEY=sk-... uv run python bench/run_bench.py --model gpt-4o-mini
uv run python bench/run_bench.py --base-url http://localhost:8000/v1 --model Qwen/Qwen3-4B-Instruct-2507

Downstream evaluation. bench/downstream.py tests whether a model trained on the data gets better. It writes a held-out, closed-book exam from your documents, trains LoRA adapters, then grades the base model, the Brainbrew adapter and a control adapter with confidence intervals. The procedure fits one 8 GB GPU; see docs/downstream-eval.md.

Contributing

Every great dataset starts with great contributors. Jump in — the cauldron is warm.

Love it? Want to make it even cooler?

1. Fork it
2. Make changes (we love clean PRs)
3. Run uv run pytest and make sure everything passes. If you change dependencies, run uv lock and regenerate the requirements*.txt exports (CI checks they match)
4. Submit PR

Ideas welcome: RAG retrieval, multi-modal support, web UI for cloud, additional export formats, etc.

License

MIT — do whatever you want. Just don't blame us if your model becomes too powerful and takes over the world.

Now go brew some brains.

Made with chaos, coffee, and zero patience for bad datasets.

Star the repo if it saved you 20 hours this week. You know you want to.

Yog-Sotho ❤️