tinyforge

Train, evaluate and serve small LLMs from scratch on modest GPUs (developed against a 4 GB RTX 3050 Ti). One engine, three front doors: CLI (for developers/CI), REST API, and a web UI (which can later be wrapped as a desktop app with Tauri/Electron without changing the backend).

Status: early alpha (0.1). Tested on Windows with one NVIDIA GPU; Linux passes the test suite in Docker but has not been run on a real GPU box. Install: pip install tinyforge, then tinyforge doctor to see what else you need (PyTorch, pip install "tinyforge[finetune]", and optional pieces such as Soup and llama.cpp).

Quick start

pip install torch --index-url https://download.pytorch.org/whl/cu124   # or /cpu
pip install -e ".[dev]"            # add ,triton for fused kernels
tinyforge doctor                   # hardware + warnings, with fixes
tinyforge pipeline --preset micro --steps 2000   # doctor -> data -> plan -> train -> eval
tinyforge generate "ROMEO:" --int8
tinyforge serve                    # UI at http://127.0.0.1:8000

What you get

Fine-tuning platform commands

Verified on one RTX 3050 Ti 4 GB laptop (Windows): a 3B and an 8B model fine-tuned through the job spec and served locally; see benchmarks/ and docs/soup-backend.md. Early alpha: single GPU, Windows tested only.

Model: decoder-only transformer with RoPE, RMSNorm, SwiGLU, tied embeddings, PyTorch SDPA (FlashAttention kernels). Presets: nano ~1M, micro ~12M, small ~28M, base ~100M.

Fine-tune a pretrained model (LoRA / QLoRA)

The most useful thing a small GPU can do. Default: SmolLM2-360M-Instruct on an instruction dataset.

pip install -e ".[finetune]"
tinyforge ft pipeline --steps 150          # doctor -> data -> plan -> train -> eval + merge
tinyforge ft generate "Explain hash tables" # tuned model;  add --base to compare with the original

Measured on an RTX 3050 Ti Laptop (4 GB), SmolLM2-360M, 150 steps x 16 examples: 2.4 GB peak VRAM, ~830 tok/s, ~10 min, held-out loss 1.314 -> 1.262 (-4.0%), no forgetting; exports a 690 MB standalone merged model. Lessons baked into the planner (all measured, see finetune.plan): small micro-batches are launch-bound (bs8 is 2.3x the tokens/s of bs4), LoRA dropout 0 is ~40% faster, and allocator-level estimates miss cuBLAS workspace, so training recovers from memory errors by shrinking the token budget instead of crashing.

Layout

src/tinyforge/   engine (config, data, model, train, evaluate, infer, kernels/), cli.py, server.py, ui/
tests/           unit + end-to-end smoke (synthetic data, no network)
infra/terraform/ S3 artifacts, IAM, no-inbound GPU spot worker (SSM access), budget alarm
.github/         CI: lint, test matrix, terraform validate, docker build, self-hosted GPU e2e
Dockerfile       CUDA runtime image

Cloud (AWS)

cd infra/terraform
terraform init -backend-config="bucket=<state-bucket>" -backend-config="key=tinyforge/tf.tfstate"
terraform apply -var enable_gpu_worker=true -var alert_email=you@example.com
# then run the `connect` output command to port-forward the UI over SSM (no open ports)

Verified results (what was actually run)

Everything below was run on one machine: Windows 11, RTX 3050 Ti Laptop GPU (4 GB), 16 GB RAM. Each row links to the raw numbers. These are single runs on small test sets, not benchmarks: read the caveats in each JSON file.

What these results do not show: general-purpose quality gains (the SQL tests use templated questions), results on other hardware or operating systems, multi-GPU training, or the standard benchmark runner (bench run, built on lm-evaluation-harness), which has not been run end to end yet.

The dashboard while serving a 3B model. KV-cache use grows linearly with context (36 KiB per token) while GPU memory stays flat, because llama.cpp reserves the cache up front.

Honest limits / roadmap

• JAX: there is a probe and a backend slot, but training is PyTorch-only today. JAX has no native Windows GPU support (use WSL2/Linux), and a second backend should be added behind the same train() interface.
• Triton: one fused kernel (RMSNorm forward, inference only). Next: fused SwiGLU, then a backward pass. Triton on Windows needs the triton-windows package.
• From-scratch models are for learning/prototyping; use fine-tuning for anything useful. A 4% held-out gain on a general instruction set is modest by design: LoRA on a small, already-tuned model mostly shifts style. Use a task-specific dataset to see larger gains.
• 4-bit QLoRA is verified on real hardware (see ROADMAP.md): it works but ends ~4.7% worse in loss than fp16 on a model that fits in fp16. Its value is fitting larger models: an 8B model trains on a 4 GB GPU via backend: soup.
• Generation speed (HF generate, eager) is ~10 tok/s on this GPU; the merged model is the thing to export to llama.cpp/vLLM for serving.
• Single-GPU, single-job (a SQLite job queue exists). Multi-GPU (FSDP/DDP) is not built and untested.
• The API requires TINYFORGE_API_TOKEN (bearer) and the server binds to localhost by default; the static UI page itself is not authenticated. Put it behind TLS, SSM/VPN or a reverse proxy before exposing it.