slmify 0.2.0.dev0

Point it at your LLM logs. See exactly where you're overpaying — fine-tune a small model, cache, or downgrade.

 SmaLLM

slmify

Find the 40-70% of your LLM traffic a small model could handle.
Prove it before you train anything. Your data never leaves your machine.

_{Open-sourced and maintained by the team at SmaLLM}

Quickstart · How it works · Benchmarks · Roadmap

---

Why this exists

Somewhere in your traffic, a frontier model is formatting JSON, filling the same template for the ten-thousandth time, and answering the same twelve questions. You are paying reasoning-model prices for work a 4B model could do faster, cheaper, and on hardware you own.

NVIDIA's research estimates that 40-70% of LLM calls in agentic systems are SLM-replaceable. They sketched the conversion pipeline, then left the measurement as an exercise for the reader.

slmify measures that number on your traffic. It ingests your logs, groups them into workloads, scores each one for SLM-fit, prices the opportunity in dollars with break-even math, recommends the cheapest safe fix, and exports training-ready datasets for whichever fine-tuning stack you already use.

Who's "we"? We're the team behind SmaLLM, a platform that runs this whole loop end to end: observe production traffic, fine-tune small models automatically, evaluate them, and canary-route them behind an OpenAI-compatible gateway. The analysis layer answers the question every team asks first ("what's actually worth fine-tuning, and what would it save?"), so we carved it out of the platform and open-sourced it. slmify is fully standalone, no strings attached: it works whether or not you ever touch the rest of our stack.

⚡ Sixty seconds to your number

If you use Claude Code, Codex CLI, or Gemini CLI, a corpus of real agent traffic is already sitting on your laptop. slmify turns it into a verdict with zero setup:

uvx slmify        # no install, no config, no API key, no network

No agent traces? Run the bundled demo corpus:

uvx slmify demo   # full report in ~30 seconds, works on a plane

Example output (illustrative):

  WORKLOADS   14 discovered · 31 days · 412,806 calls · $9,110/mo observed spend

  BAND  CLUSTER                       CALLS/MO   SPEND/MO   ACTION           EST. SAVINGS/MO
  A     c07 invoice-json-extraction    41,220     $2,410    FINE_TUNE_SLM    $1,700 - $2,100
  A     c03 support-macro-rewrite      88,410     $1,930    FINE_TUNE_SLM    $1,310 - $1,760
  B     c11 order-status-lookup        64,204       $880    EXACT_CACHE        $610 -   $780
  C     c02 code-review-comments        9,410     $1,240    DOWNGRADE_TRIAL    $340 -   $560
  ────────────────────────────────────────────────────────────────────────────────────────
  KEEP  7 clusters                    209,562     $4,980    KEEP             $0  (and that's fine)

Yes, it will tell you KEEP. A tool that recommends fine-tuning everything is a tool you can't trust.

🔍 From verdict to fine-tuned model in five commands

# 1. Analyze production logs (format auto-detected; CSV needs a --map)
uvx slmify analyze ./logs.jsonl --html report.html
uvx slmify analyze ./logs.csv --map "prompt=question,response=answer,model=model_name"

# 2. Prove feasibility BEFORE spending anything: zero training, k-shot,
#    any OpenAI-compatible endpoint (Ollama shown)
slmify probe --clusters top3 --endpoint http://localhost:11434/v1 --model qwen3:4b-instruct

# 3. Export train/eval JSONL + ready-to-run config stubs
slmify export c07        # Unsloth · Axolotl · TRL · OpenAI FT · vLLM-router

# 4. Or train locally on the same recipe ([train] extra, CUDA/Unsloth)
slmify train c07

# 5. Verify the fine-tune against the untainted holdout
slmify eval c07 --endpoint http://localhost:8000/v1 --model tuned --baseline-model base

📊 What the report gives you

For every discovered workload:

• SLM-fit band (A "Prime" → D "Poor fit") from 10 published, documented metrics: output contraction, schema stability, task-type priors, template mass, semantic cohesion, reasoning depth, and more. Top-3 evidence bullets and a confidence tag on every band. Raw numbers live in analysis.json, not the headline.
• Probe evidence on demand. slmify probe runs a candidate SLM against 30 held-out samples per cluster (k-shot, zero training) through any OpenAI-compatible endpoint and adjusts the band ±1 step. Graders run cheapest-first (exact → JSON-schema → embedding similarity); an LLM judge only runs if you supply a judge endpoint, and flat judge scores automatically switch to ranking mode.
• Measured spend and savings as ranges, {low, point, high} with every assumption printed, priced from a vendored LiteLLM pricing snapshot, monthly volumes extrapolated with a bootstrap CI over your observed days. A 3-day log honestly yields wide bars.
• Dedicated-GPU break-even math. The report tells you the sustained calls/hour where a dedicated L4/A10G/A100 beats your current spend, and says "use serverless" when you're below it.
• An action stack, decided by auditable ordered rules you can override with --policies: FINE_TUNE_SLM, DOWNGRADE_TRIAL, EXACT_CACHE, SEMANTIC_CACHE, PROMPT_CACHE / TRIM, FIX_QUALITY_FIRST (with a named remedy), or an honest KEEP.
• Cache economics as floor / realistic / ceiling. Exact-duplicate rate is the zero-risk floor; semantic near-duplicate mass is the ceiling.
• Episode reconstruction. Multi-turn conversations in flat logs are rebuilt with conservative prefix matching, so volumes, duplicate rates, and dataset exports never double-count growing message arrays.

slmify export <cluster> writes train.jsonl / eval.jsonl in OpenAI chat format, plus config stubs. The eval set is a stratified holdout never included in training data, so your fine-tune gets verified on untainted samples.

⚙️ How it works

flowchart LR
    A["📥 Any logs<br/>Claude Code · Codex · Gemini CLI<br/>Langfuse · OpenAI JSONL · CSV"] --> B["🧵 Episode<br/>reconstruction"]
    B --> C["🧩 Template mining<br/>(Drain3)"]
    C --> D["🗺️ Semantic clustering<br/>(local embeddings)"]
    D --> E["🎯 10-metric<br/>SLM-fit scoring"]
    E --> F["💰 Savings + break-even<br/>+ action stack"]
    F --> G["🔬 Probe<br/>(zero-training proof)"]
    G --> H["🚀 Export / Train / Eval<br/>Unsloth · Axolotl · TRL · OpenAI"]

Workloads are grouped on three levels (workflow step → prompt template → semantic cluster), so "the same call with different variables" and "different calls that mean the same thing" both land where they belong.

🧭 Built different

• 🔒 Local-first, zero telemetry, zero required API keys. Your prompts never leave your machine. The default embedder is a 30 MB static model; the tool runs offline.
• 🧮 No false precision, ever. Bands not decimals, ranges not points, every assumption printed. Expect frequent, honest KEEP recommendations.
• 🔬 Evidence over vibes. Every score ships with the exact metrics behind it. Every formula is documented and user-overridable via --policies.
• 🇨🇭 Neutral. Reads every vendor's logs, exports to every trainer. No login, no gates, no nags.

🔌 Works with your stack

Reads (inputs)	Feeds (outputs)
Claude Code traces (~/.claude/projects)	Unsloth config + dataset
Codex CLI sessions (~/.codex/sessions)	Axolotl config + dataset
Gemini CLI sessions (~/.gemini/tmp)	TRL training script stub
Langfuse observation exports	OpenAI fine-tuning JSONL
OpenAI-style JSONL (3 dialects)	vLLM router config stub
Generic JSONL/CSV via dotted-path --map	HTML + JSON reports
LiteLLM (drop-in SlmifyLogger callback)

Adapters are pluggable via the slmify.adapters entry-point group. OTel GenAI lands in v0.3.

🥊 How slmify compares

	slmify	OpenAI Distillation	Braintrust / LangSmith	Kiln
Finds what to fix in your traffic	✅ scored workloads	✗	clusters, no scoring/savings	task-first
Cross-vendor (reads any logs)	✅	OpenAI only	their platform	n/a
Runs locally, data never leaves	✅	✗	✗	✅
Proves SLM feasibility pre-training	✅ probe	✗	✗	✗
$ savings with break-even math	✅ ranges	✗	✗	✗
Exports to any trainer	✅	✗	partial	✅

As of 2026-07. These tools solve adjacent problems; several pair well with slmify rather than replacing it.

📈 The market is moving

"I believe the future of agentic AI is many, many, many small language models."

Andy Markus, Chief Data Officer, AT&T (Feb 2026)

AT&T rebuilt its stack around fine-tuned SLMs at 27B tokens/day and publicly reported cost reductions up to 90%. NVIDIA's position paper (arXiv:2506.02153) argues SLMs are the future of agentic AI and that most agent calls don't need a frontier model.

They built the measurement and conversion by hand, with ML teams. slmify gives you the measurement in one command.

🧪 Benchmarks

benchmarks/ holds reproducible runs:

• Ground-truth validation of the whole grouping pipeline on the bundled labeled corpus: purity 1.00, V-measure 0.94.
• One-command public-data benchmark (WildChat-1M / LMSYS-Chat-1M) measuring the SLM-able share of real traffic against NVIDIA's 40-70% agentic estimate.

📦 Install

uv tool install slmify        # or: pipx install slmify / pip install slmify

Core install is light: polars, numpy, scikit-learn, model2vec, drain3, pydantic, typer, rich, httpx, pyyaml. No torch, no database, no compiled clustering libs. Extras: [st] (sentence-transformers embedders), [train] (local LoRA fine-tuning).

Offline degradation ladder: demo needs zero network → the first real analysis downloads the 30 MB default embedder once (then cached) → fully offline, use --embedder hash (TF-IDF, already installed). Degraded clusters, still useful, never a dead end.

🔐 Privacy posture

Report exemplars are masked by default (<EMAIL>, <NUM>, <ID>, …) via the same template miner that names your clusters; --show-raw disables. Dataset exports contain raw text (training needs real data) and say so loudly. There is no telemetry of any kind.

🗺️ Roadmap

• v0.1 — analyzer, HTML report, dataset exports
• v0.2 — zero-training probe, local LoRA trainer, eval harness, LiteLLM callback
• v0.3 — --watch/CI diff mode · OTel GenAI adapter · in-process adapter eval · open calibration study on public data · plugin authoring docs · --backend mlx (Apple Silicon training)

Have an opinion on what comes next? Open a discussion.

⭐ Support the project

If slmify found money in your logs, or told you honestly that there was none to find, star the repo. It's how other engineers discover it, and it directly shapes how much time we can spend on v0.3.

🤝 Contributing

Bug reports, adapter plugins (slmify.adapters entry points), and benchmark submissions are all welcome. Check good first issue to get started, and read CONTRIBUTING.md for the dev setup.

❓ FAQ

Does my data ever leave my machine?

No. There is no telemetry, no login, and no required API key. The only network calls slmify ever makes are ones you explicitly configure: downloading the default embedder once, slmify pricing --update, and probe/eval calls to endpoints you point it at (which can be localhost).

Do I need a GPU?

Not for analysis, probing, or exports. Only slmify train (the optional [train] extra) needs CUDA. Everything else runs on a laptop.

Why does it keep telling me KEEP?

Because most traffic genuinely isn't worth fine-tuning, and pretending otherwise would make the A-band verdicts worthless. KEEP means the math didn't clear the bar: volume too low, cohesion too weak, or savings below break-even. That's the tool working.

How accurate are the savings numbers?

They're ranges, not promises. Prices come from a vendored LiteLLM pricing snapshot, volumes are extrapolated with a bootstrap CI over your observed days, and every assumption is printed in the report. Short logs produce wide bars on purpose.

Can I change how workloads are scored?

Yes. Every formula and threshold is documented, and --policies accepts a YAML override for the scoring weights and the action-stack rules.

Is this just a funnel for a paid product?

It's a real standalone tool first. We (the SmaLLM team) sell a platform that automates what slmify recommends: training, serving, and canary-routing the small models. slmify never requires it, never phones home, and exports to every trainer, including ones that compete with us. If you never visit our site, slmify still does its entire job.

📄 License

Apache-2.0.

<sub>slmify is built and maintained by the team behind SmaLLM,
the platform that closes the loop slmify opens: automated fine-tuning, evaluation, and canary routing of the workloads it finds.

slmify stays free, local, and neutral either way. Built by engineers who got tired of guessing.</sub>