soup-cli 0.72.4

Fine-tune and post-train LLMs in one command. No SSH, no config hell.

Soup

Fine-tune and post-train LLMs in one command. No SSH, no config hell.

Website · Quick Start · Config · Docs · Commands · Models · Discord

---

Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done.

pip install "soup-cli[train]"   # add [train] to fine-tune; bare `soup-cli` is the light CLI
soup init --template chat
soup train

Why Soup?

Training LLMs is still painful. Even experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. Soup fixes that.

• Zero SSH. Never SSH into a broken GPU box again.
• One config. A simple YAML file is all you need.
• Auto everything. Batch size, GPU detection, quantization — handled.
• Works locally. Train on your own GPU with QLoRA. No cloud required.

What's New

v0.72.4 — align on a laptop: DPO, ORPO, SimPO and KTO over layer streaming. Layer streaming keeps the frozen base out of VRAM and feeds it to the GPU one decoder layer at a time. It used to support supervised fine-tuning only; now it runs the preference losses too.

• DPO's reference model is free. DPO needs a reference to compare against, and a second copy of the model would double memory and defeat the whole point. Soup uses the same streamed base with its adapters switched off — one set of weights, one stream. Measured on an RTX 3050 4 GB: streamed DPO peaked at 0.914× the supervised-fine-tuning peak. Forcing a real second model in the same test cost +730 MB — exactly one copy of the weights.
• KTO is not reference-free, however it is usually described: it picks its reference the same way DPO does, so it gets the same treatment. ORPO and SimPO genuinely are.
• Bit-exact against a normal, non-streamed run of the same loss — 0.0 difference, the bar every release in this series has to clear.
• The VRAM pre-flight knows a paired loss is twice the rows, because chosen and rejected go through the model as one tensor.
• Honest cost: the reference is free in memory, not in time — DPO reads the layer stack 1.52× as often per step as supervised fine-tuning does.
• grpo / ppo stay excluded on purpose: generation re-reads every layer per token, which is exactly what streaming cannot amortise.
• Still BETA.

# soup.yaml — then just `soup train --config soup.yaml`
training:
  stream_layers: true      # base streams out of VRAM; only the adapter trains
  quantization: 4bit       # NF4 — ~4x smaller store, so 8B fits a 4 GB card
  batch_size: 4            # v0.72.3: bigger batches amortise the weight read
  stream_source: auto      # RAM when it fits, NVMe disk when it does not

Trained with stream_layers: true on v0.72.0? That adapter is inert — its tensors were saved under keys with an extra .inner. segment, so every loader returned the untuned base. Fixed in v0.72.1; re-run or re-save. Check with: python -c "from safetensors.torch import load_file; print([k for k in load_file('adapter_model.safetensors') if '.inner.' in k][:3])"

Previous release — v0.71.40, soup reward synth (generate a reward verifier from your data)

Point soup reward synth at a JSONL of reference outputs and it infers a deterministic verifier, writes a readable / committable .py reward function, and — the part nobody else does — refuses to emit one that can't tell your references from bad answers (four families: numeric / json_schema / regex / tool_call; a mandatory calibration report is the moat). Reward ensembles (reward_fn: "accuracy,format") also train now. (#311)

soup reward synth references.jsonl -o reward.py --output-report calib.json

Previous release — v0.71.39, CI for weights not prompts (emit + provenance-bind the ship verdict)

soup ship's verdict became emittable, committable, and provenance-bound: --emit-evidence makes a run replay into an identical verdict, eval.ship in soup.yaml + --config makes the gate policy reviewable, and --config binds evidence to the exact recipe that produced it (stale evidence → exit 3). soup ship --push owner/repo#N posts the SHIP / DON'T-SHIP card on the PR.

Previous release — v0.71.38, The gate grows teeth (real leg-2 regression gate)

soup ship's regression leg became real: a fixed, extraction-based scorer over seven bundled, offline suites (MCQ · arithmetic · tool-calling · JSON validity · safety/refusal). A tune that wins your task but quietly breaks tool-calling now gets a DON'T SHIP. Zero new deps.

soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error

Previous release — v0.71.33, soup draft (measure speculative decoding)

soup draft measure reports a draft model's acceptance rate + real plain-vs-assisted tok/s (exit 0/2/1 for CI); soup draft distill distils your target into a dense tiny draft, auto-wired into soup serve --auto-spec. The honest result on a small same-family pair: distillation didn't move acceptance (69.3% → 69.3%) and assisted decoding was a net slowdown — which is exactly the number you want before shipping speculative decoding.

soup draft measure --target ./my-tuned-model --draft HuggingFaceTB/SmolLM2-135M-Instruct \
  --prompts prod-prompts.jsonl        # -> acceptance %, real tok/s, ship-or-not

Full history: CHANGELOG.md · GitHub Releases.

Quick Start

1. Install

# Light core: CLI + config + data tools, no PyTorch
pip install soup-cli

# Add the training stack (torch, transformers, peft, trl, datasets, …)
pip install "soup-cli[train]"

# Everything (train + serve + ui + data) in one shot
pip install "soup-cli[all]"

# Or from GitHub (latest dev)
pip install git+https://github.com/MakazhanAlpamys/Soup.git

The full extras table (fast, mlx, serve, eval, ui, vision, audio, …) lives in docs/models.md.

Use double quotes around the extra. They are the only spelling that works in every shell — cmd.exe, PowerShell, bash, and zsh.

Older tutorials and videos (including some of ours) show the single-quoted pip install 'soup-cli[train]'. That is bash / zsh / PowerShell syntax, and it fails on Windows cmd.exe, which has no single-quote quoting and hands the quotes straight to pip:

ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at the start of dependency specifier

If you hit that, swap the ' for " — pip is rejecting a literal quote character, nothing is wrong with the package. (Dropping the quotes entirely works on Windows too, but zsh then reads [train] as a glob and fails.)

soup init, soup data …, and the other data/inspection commands work on the light install. Fine-tuning (soup train) needs the [train] extra.

2. Create a config

soup init                       # interactive wizard
soup init --template chat       # or start from a template

Templates: chat, code, tool-calling, medical, reasoning, vision, kto, orpo, simpo, ipo, bco, rlhf, pretrain, moe, longcontext, embedding, audio.

3. Train, test, ship

soup train --config soup.yaml                 # LoRA, quantization, batching — all handled
soup chat  --model ./output                    # talk to your model
soup push  --model ./output --repo you/my-model

soup merge  --adapter ./output                              # merge LoRA into the base
soup export --model ./output --format gguf --quant q4_k_m   # GGUF for Ollama / llama.cpp

More export targets (ONNX, TensorRT, AWQ, GPTQ, BitNet) and deployment options live in docs/serving-and-export.md.

Configuration

A complete soup.yaml:

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
# backend: unsloth  # 2-5x faster, pip install "soup-cli[fast]"

data:
  train: ./data/train.jsonl
  format: alpaca
  val_split: 0.1

training:
  epochs: 3
  lr: 2e-5
  batch_size: auto
  lora:
    r: 64
    alpha: 16
  quantization: 4bit

output: ./output

config/schema.py is the single source of truth for every field. Advanced data, training, and PEFT options are documented under Documentation.

Documentation

The full feature reference lives in docs/. Start here:

Guide	Covers
Training tasks & methods	SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors
PEFT, long context & efficiency	DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning
Performance & quantization	QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, layer streaming, multi-GPU / DeepSpeed / FSDP
Data engineering	Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs
Evaluation & probes	Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, soup advise
Serving & export	OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge
Adapters, registry & governance	Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (soup loop), knowledge editing, steering, supply-chain controls (scan/sign/BOM/attest/audit/airgap)
Compliance & governance quickstart	HIPAA/SOC2/EU-AI-Act/SR-11-7 init templates, provenance (BOM/attest/repro-receipt), audit log, air-gap, model-card autogen (soup card), CI gate (soup ci init)
Backends, platform & ops	MLX/Unsloth backends, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands
Command reference	The full soup command list
Supported models & extras	Recommended model families, the VRAM size guide, the pip extras matrix

Data Formats

All formats are auto-detected from JSONL, JSON, CSV, Parquet, or TXT:

• alpaca — {"instruction": ..., "input": ..., "output": ...}
• sharegpt — {"conversations": [{"from": "human", "value": ...}, ...]}
• chatml — {"messages": [{"role": "user", "content": ...}, ...]}
• dpo / orpo / simpo / ipo — {"prompt": ..., "chosen": ..., "rejected": ...}
• kto — {"prompt": ..., "completion": ..., "label": true}
• llava / sharegpt4v (vision), audio, plaintext (pre-training), embedding, prm, pre_tokenized, video, multimodal

Full schemas and the Axolotl/LlamaFactory-parity data pipeline (remote URIs, streaming, sharding, interleaving, vocab expansion, document ingestion) are in docs/data.md.

Common Commands

soup train  --config soup.yaml        # train (SFT/DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/...)
soup infer  --model ./output --input prompts.jsonl   # batch inference
soup chat   --model ./output          # interactive chat
soup serve  --model ./output          # OpenAI-compatible API server
soup merge  --adapter ./output        # merge LoRA into the base model
soup export --model ./output --format gguf           # export for deployment
soup eval   benchmark --model ./output               # evaluate
soup data   inspect ./data/train.jsonl               # dataset stats
soup recipes list                     # 100+ ready-made model recipes
soup autopilot --model <id> --data d.jsonl --goal chat  # zero-config
soup doctor                           # check GPU / deps / environment

The complete command list is in docs/commands.md.

Supported Models

Soup works with any text-generation model on the HuggingFace Hub — if it loads with AutoModelForCausalLM, it works, zero config changes. Llama 3.x/4, Qwen 2.5/3, Gemma 3, Mistral, Mixtral, DeepSeek R1/V3, Phi-4, and 100+ others ship as ready-made recipes (soup recipes list).

VRAM	Max model (QLoRA 4-bit)	Example
8 GB	~7B	Llama-3.1-8B, Mistral-7B
16 GB	~14B	Phi-4-14B, Qwen2.5-14B
24 GB	~34B	CodeLlama-34B, Yi-1.5-34B
48 GB	~70B	Llama-3.3-70B
80 GB+	70B+ (full) or MoE	Mixtral-8x22B, DeepSeek-V3

Full model + vision tables and the optional-extras matrix are in docs/models.md.

Docker

Run Soup without installing CUDA or PyTorch locally (image published to GHCR on every release):

docker pull ghcr.io/makazhanalpamys/soup:latest
docker run --gpus all -v $(pwd):/workspace ghcr.io/makazhanalpamys/soup train --config soup.yaml
docker compose up   # or build locally

Requirements

• Python 3.10+
• GPU with CUDA (recommended), Apple Silicon (MPS), or CPU (experimental — very slow)
• 8 GB+ VRAM for 7B models with QLoRA

All training tasks run on CPU for testing (quantization auto-disabled). Optional extras (train, all, fast, vision, qat, serve, serve-fast, ui, eval, deepspeed, liger, mlx, onnx, tensorrt, …) are listed in docs/models.md.

Troubleshooting

soup doctor    # GPU, system resources, dependencies, and version in one place

• ImportError: DLL load failed while importing _C (Windows) — reinstall PyTorch for your CUDA version: pip install torch --index-url https://download.pytorch.org/whl/cu121.
• soup version ≠ pip show soup-cli — multiple Python installs; use a virtualenv.

Development

git clone https://github.com/MakazhanAlpamys/Soup.git
cd Soup
pip install -e ".[dev]"

ruff check src/soup_cli/ tests/    # lint
pytest tests/ -v                   # unit tests (fast, no GPU)
pytest tests/ -m smoke -v          # smoke tests (downloads a tiny model, trains)

pre-commit install                 # optional: ruff lint+format on commit

See CONTRIBUTING.md for the full workflow and SECURITY.md to report a vulnerability.

Support Soup

Soup is Apache-2.0 and free — and stays that way. It is built and maintained in the open on a single 4 GB laptop, which is why every performance number in these docs is measured rather than claimed.

If Soup saved you a training run, starring the repo helps most, and it costs nothing.

The next most useful thing is hardware. Multi-GPU, 8B+ validation, and Apple Silicon are the parts a single 4 GB laptop cannot reach, so they ship behind honest "requires " gates instead of unverified claims. If you have access to a bigger box — or GPU credits going unused — running one of the help wanted issues and posting the numbers moves Soup further than anything else. Those issues say exactly what is blocked on hardware today.

Contributors

Built by the community ❤️ — thank you to everyone who has contributed. See CONTRIBUTORS.md.

Contact

Bugs and feature requests belong in the issue tracker, questions in Discussions — both get answered faster and help the next person with the same problem.

For live chat, setup help, and everything that reads better as a conversation, join the Discord. Anything that should still be findable in six months belongs in Issues or Discussions — a Discord answer helps one person, an issue helps everyone who hits the same thing. The Code of Conduct applies there too.

For anything that does not fit in public — security reports (see SECURITY.md), Code of Conduct matters, or press — email [e-mail]. That is the project address and the right one for anything Soup-related. [e-mail] is the maintainer's personal address; it reaches the same person and is a fine fallback.

Citing Soup

Layer streaming — training an 8B model on a 4 GB laptop GPU by streaming the frozen base from host RAM one decoder layer at a time — is described in a preprint, together with the correctness protocol that verifies a streamed run is bit-exact against a resident one:

Makazhan, A. (2026). Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU. Zenodo. https://doi.org/10.5281/zenodo.21771064

The measurement records behind every number in it are in benchmarks/, published as written — including the failures, the assumptions that turned out wrong, and the numbers that were measured and then discarded.

@misc{makazhan2026exact,
  title        = {Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU},
  author       = {Makazhan, Alpamys},
  year         = {2026},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.21771064},
  url          = {https://doi.org/10.5281/zenodo.21771064}
}

License

Apache-2.0. Copyright © the Soup contributors.