soup-cli 0.71.38

Fine-tune and post-train LLMs in one command. No SSH, no config hell.

Soup

Fine-tune and post-train LLMs in one command. No SSH, no config hell.

Website · Quick Start · Config · Docs · Commands · Models

---

Soup turns the pain of LLM fine-tuning into a simple workflow. One config, one command, done.

pip install "soup-cli[train]"   # add [train] to fine-tune; bare `soup-cli` is the light CLI
soup init --template chat
soup train

Why Soup?

Training LLMs is still painful. Even experienced teams spend 30-50% of their time fighting infrastructure instead of improving models. Soup fixes that.

• Zero SSH. Never SSH into a broken GPU box again.
• One config. A simple YAML file is all you need.
• Auto everything. Batch size, GPU detection, quantization — handled.
• Works locally. Train on your own GPU with QLoRA. No cloud required.

What's New

v0.71.38 — The gate grows teeth. soup ship answers the one question that matters after a fine-tune — did my task improve WITHOUT silently breaking everything else? This release makes the "without breaking everything else" half real.

• A gate that actually catches regressions. The regression leg used to be 15 trivia prompts scored by raw substring — it credited "B" for "Berlin", "3" for "13", and had zero items for tool-calling, safety, or JSON. Now it's a fixed, extraction-based scorer over seven bundled, offline suites (MCQ · arithmetic · tool-calling · JSON validity · safety/refusal). A tune that WINS your task but quietly breaks tool-calling now gets a DON'T SHIP — the old gate waved it through.
• Zero new dependencies, no network. Every suite ships in the wheel and is scored by the pure scorers Soup already has — no lm-eval, no download. soup ci init's core-only gate keeps working.
• Distinct exit codes for CI. Usage errors moved off exit 2 to 3, so 2 now means only DON'T-SHIP — a typo'd flag is no longer indistinguishable from a caught regression.
• Breaking, on purpose. The fixed scorer can change an existing run's verdict — because the old one was reporting false negatives. Recompute any committed --baseline.

# leg 1 (task win) AND leg 2 (no regression on the bundled suite) -> one verdict
soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error

soup ship --evidence scores.json    # offline verdict from pre-computed scores

Previous release — v0.71.36, Data Moat II (semantic dedup · topics · canaries · replay)

soup data dedup --semantic dedups by meaning; soup data topics shows a coverage table; soup data canary insert|check proves whether a model memorized inserted secrets (exit 2 on a leak); soup train --replay old.jsonl interleaves old data so a new task doesn't erase it.

soup data topics train.jsonl
soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
soup train --config sft.yaml --replay old_task.jsonl --replay-ratio 0.1

Previous release — v0.71.33, soup draft (measure speculative decoding)

soup draft measure reports a draft model's acceptance rate + real plain-vs-assisted tok/s (exit 0/2/1 for CI); soup draft distill distils your target into a dense tiny draft, auto-wired into soup serve --auto-spec. The honest result on a small same-family pair: distillation didn't move acceptance (69.3% → 69.3%) and assisted decoding was a net slowdown — which is exactly the number you want before shipping speculative decoding.

soup draft measure --target ./my-tuned-model --draft HuggingFaceTB/SmolLM2-135M-Instruct \
  --prompts prod-prompts.jsonl        # -> acceptance %, real tok/s, ship-or-not

Full history: CHANGELOG.md · GitHub Releases.

Quick Start

1. Install

# Light core: CLI + config + data tools, no PyTorch
pip install soup-cli

# Add the training stack (torch, transformers, peft, trl, datasets, …)
pip install "soup-cli[train]"

# Everything (train + serve + ui + data) in one shot
pip install "soup-cli[all]"

# Or from GitHub (latest dev)
pip install git+https://github.com/MakazhanAlpamys/Soup.git

The full extras table (fast, mlx, serve, eval, ui, vision, audio, …) lives in docs/models.md.

Use double quotes around the extra. They are the only spelling that works in every shell — cmd.exe, PowerShell, bash, and zsh.

Older tutorials and videos (including some of ours) show the single-quoted pip install 'soup-cli[train]'. That is bash / zsh / PowerShell syntax, and it fails on Windows cmd.exe, which has no single-quote quoting and hands the quotes straight to pip:

ERROR: Invalid requirement: "'soup-cli[train]'": Expected package name at the start of dependency specifier

If you hit that, swap the ' for " — pip is rejecting a literal quote character, nothing is wrong with the package. (Dropping the quotes entirely works on Windows too, but zsh then reads [train] as a glob and fails.)

soup init, soup data …, and the other data/inspection commands work on the light install. Fine-tuning (soup train) needs the [train] extra.

2. Create a config

soup init                       # interactive wizard
soup init --template chat       # or start from a template

Templates: chat, code, tool-calling, medical, reasoning, vision, kto, orpo, simpo, ipo, bco, rlhf, pretrain, moe, longcontext, embedding, audio.

3. Train, test, ship

soup train --config soup.yaml                 # LoRA, quantization, batching — all handled
soup chat  --model ./output                    # talk to your model
soup push  --model ./output --repo you/my-model

soup merge  --adapter ./output                              # merge LoRA into the base
soup export --model ./output --format gguf --quant q4_k_m   # GGUF for Ollama / llama.cpp

More export targets (ONNX, TensorRT, AWQ, GPTQ, BitNet) and deployment options live in docs/serving-and-export.md.

Configuration

A complete soup.yaml:

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
# backend: unsloth  # 2-5x faster, pip install "soup-cli[fast]"

data:
  train: ./data/train.jsonl
  format: alpaca
  val_split: 0.1

training:
  epochs: 3
  lr: 2e-5
  batch_size: auto
  lora:
    r: 64
    alpha: 16
  quantization: 4bit

output: ./output

config/schema.py is the single source of truth for every field. Advanced data, training, and PEFT options are documented under Documentation.

Documentation

The full feature reference lives in docs/. Start here:

Guide	Covers
Training tasks & methods	SFT, DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/BCO, tool-calling, PRM, pre-training, distillation, classification, vision/audio/TTS, unlearning, RAFT/RA-DIT, loop-hardening detectors
PEFT, long context & efficiency	DoRA, LoRA+, rsLoRA, VeRA, OLoRA, NEFTune, PiSSA, ReLoRA, optimizer & PEFT zoo, LLaMA Pro, GaLore, YaRN/LongLoRA, packing, curriculum, auto-tuning
Performance & quantization	QAT, FP8, Quant Menu (I + II), KV-cache, NVFP4, save formats, Cut Cross-Entropy, gradient checkpointing, kernels, activation offloading, multi-GPU / DeepSpeed / FSDP
Data engineering	Formats, the Axolotl/LF-parity pipeline, data tools, synthetic generation & forge, quality scorecards, trace tooling, remote datasets, mixing, recipe DAGs
Evaluation & probes	Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, soup advise
Serving & export	OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge
Adapters, registry & governance	Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (soup loop), knowledge editing, steering, supply-chain controls (scan/sign/BOM/attest/audit/airgap)
Compliance & governance quickstart	HIPAA/SOC2/EU-AI-Act/SR-11-7 init templates, provenance (BOM/attest/repro-receipt), audit log, air-gap, model-card autogen (soup card), CI gate (soup ci init)
Backends, platform & ops	MLX/Unsloth backends, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands
Command reference	The full soup command list
Supported models & extras	Recommended model families, the VRAM size guide, the pip extras matrix

Data Formats

All formats are auto-detected from JSONL, JSON, CSV, Parquet, or TXT:

• alpaca — {"instruction": ..., "input": ..., "output": ...}
• sharegpt — {"conversations": [{"from": "human", "value": ...}, ...]}
• chatml — {"messages": [{"role": "user", "content": ...}, ...]}
• dpo / orpo / simpo / ipo — {"prompt": ..., "chosen": ..., "rejected": ...}
• kto — {"prompt": ..., "completion": ..., "label": true}
• llava / sharegpt4v (vision), audio, plaintext (pre-training), embedding, prm, pre_tokenized, video, multimodal

Full schemas and the Axolotl/LlamaFactory-parity data pipeline (remote URIs, streaming, sharding, interleaving, vocab expansion, document ingestion) are in docs/data.md.

Common Commands

soup train  --config soup.yaml        # train (SFT/DPO/GRPO/PPO/KTO/ORPO/SimPO/IPO/...)
soup infer  --model ./output --input prompts.jsonl   # batch inference
soup chat   --model ./output          # interactive chat
soup serve  --model ./output          # OpenAI-compatible API server
soup merge  --adapter ./output        # merge LoRA into the base model
soup export --model ./output --format gguf           # export for deployment
soup eval   benchmark --model ./output               # evaluate
soup data   inspect ./data/train.jsonl               # dataset stats
soup recipes list                     # 100+ ready-made model recipes
soup autopilot --model <id> --data d.jsonl --goal chat  # zero-config
soup doctor                           # check GPU / deps / environment

The complete command list is in docs/commands.md.

Supported Models

Soup works with any text-generation model on the HuggingFace Hub — if it loads with AutoModelForCausalLM, it works, zero config changes. Llama 3.x/4, Qwen 2.5/3, Gemma 3, Mistral, Mixtral, DeepSeek R1/V3, Phi-4, and 100+ others ship as ready-made recipes (soup recipes list).

VRAM	Max model (QLoRA 4-bit)	Example
8 GB	~7B	Llama-3.1-8B, Mistral-7B
16 GB	~14B	Phi-4-14B, Qwen2.5-14B
24 GB	~34B	CodeLlama-34B, Yi-1.5-34B
48 GB	~70B	Llama-3.3-70B
80 GB+	70B+ (full) or MoE	Mixtral-8x22B, DeepSeek-V3

Full model + vision tables and the optional-extras matrix are in docs/models.md.

Docker

Run Soup without installing CUDA or PyTorch locally (image published to GHCR on every release):

docker pull ghcr.io/makazhanalpamys/soup:latest
docker run --gpus all -v $(pwd):/workspace ghcr.io/makazhanalpamys/soup train --config soup.yaml
docker compose up   # or build locally

Requirements

• Python 3.10+
• GPU with CUDA (recommended), Apple Silicon (MPS), or CPU (experimental — very slow)
• 8 GB+ VRAM for 7B models with QLoRA

All training tasks run on CPU for testing (quantization auto-disabled). Optional extras (train, all, fast, vision, qat, serve, serve-fast, ui, eval, deepspeed, liger, mlx, onnx, tensorrt, …) are listed in docs/models.md.

Troubleshooting

soup doctor    # GPU, system resources, dependencies, and version in one place

• ImportError: DLL load failed while importing _C (Windows) — reinstall PyTorch for your CUDA version: pip install torch --index-url https://download.pytorch.org/whl/cu121.
• soup version ≠ pip show soup-cli — multiple Python installs; use a virtualenv.

Development

git clone https://github.com/MakazhanAlpamys/Soup.git
cd Soup
pip install -e ".[dev]"

ruff check src/soup_cli/ tests/    # lint
pytest tests/ -v                   # unit tests (fast, no GPU)
pytest tests/ -m smoke -v          # smoke tests (downloads a tiny model, trains)

pre-commit install                 # optional: ruff lint+format on commit

See CONTRIBUTING.md for the full workflow and SECURITY.md to report a vulnerability.

Contributors

Built by the community ❤️ — thank you to everyone who has contributed. See CONTRIBUTORS.md.

License

Apache-2.0. Copyright © the Soup contributors.