taskdistill

Replace an expensive LLM API call on a narrow task with a small fine-tuned model on your Mac, and prove it with numbers. taskdistill captures the traffic your application already sends, curates it into training data, LoRA fine-tunes a 0.5B–1.5B Qwen2.5 student with MLX on Apple Silicon, evaluates it against the original model on held-out data, and serves an OpenAI-compatible cascade: the student answers first and hands the request to the original model (the teacher) when its confidence is below a threshold chosen on validation data. Your application changes only its base URL.

Why

Many production LLM calls are narrow: route a support message to one of N intents, pull eight fields out of an invoice e-mail. They ship on a mid-tier API model because that is the fastest way to launch. At volume the call becomes a steady cost and a latency floor (in this project the teacher answered in about 594 ms at p50 and 1,242 ms at p95 (Banking77 labelling calls, 2026-09-26)), and it sends every input to a third party. A small fine-tuned model could take most of that traffic, but teams hold back for three reasons: nobody has the logged data in trainable shape, nobody trusts the small model's quality, and nobody knows which requests it will get wrong.

taskdistill is for the engineers who own such a call. It gives you a reproducible way to

1. collect training data from the traffic you already have (a capture proxy, or an import of existing logs),
2. measure the student against the teacher on held-out data, with baselines and confidence intervals, and
3. run a cascade whose quality/cost trade-off you pick explicitly, on validation data, instead of hoping for it.

How it relates to other tools

The gap taskdistill fills: one local, open-source pipeline from captured traffic to a cascade with a validation-chosen threshold on Apple Silicon, with an evaluation report you can reproduce offline. (Statements checked against each project's own site or repository on 2026-09-26.)

Quickstart

On an Apple Silicon Mac (macOS 14 or later) with uv installed. No API key is needed: without one, the demo replays the recorded teacher outputs that ship with the package.

uvx --from git+https://github.com/B0yko/taskdistill taskdistill demo banking77

Measured quick-profile demos in replay mode, each in a fresh workspace and directory (2026-09-27; reports/demo_timing.json):

The cold run (empty uv cache and Hugging Face cache on the Mac Studio, Apple M4 Max, 128 GB) took 1.7 min: 7.1 s to install and 1.6 min for the demo, including the 0.29 GB base-model download. Every measured run finished in under five minutes; slower Macs and slower connections will take longer.

The demo runs the whole pipeline on Banking77 and leaves a trained student, a report in reports/banking77/ and a request.json in the current directory. Serve the cascade and call it:

uvx --from git+https://github.com/B0yko/taskdistill taskdistill serve --task banking77

curl -s -D - http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d @request.json

The response is a standard chat.completion; the x-taskdistill-route header says whether the student or the teacher answered and x-taskdistill-confidence carries the student's confidence. taskdistill demo invoices does the same for JSON extraction on synthetic invoices. Both demos use the quick profile, sized to finish in a few minutes: Banking77 trains on 2,000 examples (300 validation, 500 test, 200 iterations) and invoices on 120 documents (24 validation, 36 test, 6 per layout, 30 iterations), cut down from 400/60/100 so the demo stays within the time budget. Use --profile full for the configuration behind the numbers below (it takes tens of minutes to hours on a laptop).

How it works

flowchart LR
  subgraph today[Your application today]
    A[App] -->|base URL| P[capture proxy]
    P --> T[(teacher API)]
  end
  P --> S[(SQLite store)]
  S --> C[curate] --> D[data/*.jsonl] --> TR[train: LoRA with MLX] --> AD[adapter]
  AD --> E[eval]
  E -->|validation: run, threshold, calibration| TH[selected_run.json + threshold.json]
  E -->|test: report only| R[report]
  subgraph cascade[Your application after]
    A2[App] -->|base URL| CS[cascade server]
    CS -->|confidence >= t| ST[student on the Mac]
    CS -->|confidence < t| T2[(teacher API)]
  end
  TH --> CS
  CS --> S
  S --> R

Confidence. Classification decodes greedily under a trie of the canonical labels, so every answer is a valid label; confidence is the product of the renormalised probabilities of the chosen tokens (the end token included, so a label that is a prefix of another is handled). Extraction generates JSON freely; each field's confidence is the product of the probabilities of the tokens spanning its value, and the document's confidence is the minimum over fields (0 for invalid JSON or a schema violation).

Selection on validation only. Every choice (base model, seed, checkpoint, threshold, isotonic calibration, baseline hyperparameters) is made by a function that accepts only a ValidationSplit; passing a TestSplit raises TypeError. The threshold is the one with the lowest escalation rate that meets the target on validation, and the test split then reports whether the target held.

Use it on your own call

Say your application classifies support tickets with a paid API model.

1. Install the command and create a task spec.

   uv tool install git+https://github.com/B0yko/taskdistill
   

   taskdistill init tickets --type classification
   

   Edit tasks/tickets/teacher_prompt.md (the system prompt your application sends today), labels.txt, and in task.yaml the teacher's model, base_url and, for OpenRouter, the pinned provider in extra_body. Everything below runs in the directory that contains tasks/; the workspace defaults to ./.taskdistill.

2. Capture traffic. Start the proxy and change only the base URL of your OpenAI client:

   export TASKDISTILL_TEACHER_BASE_URL=https://openrouter.ai/api/v1   # where the proxy forwards
   taskdistill capture --task tickets
   

   from openai import OpenAI
   
   client = OpenAI(base_url="http://127.0.0.1:8787/t/tickets/v1", api_key=YOUR_KEY)  # the key is passed through
   

   Or import logs you already have (--format openai for request/response pairs, pairs for input/output, inputs for unlabelled inputs that the teacher will label). Gold labels, if you have any, are imported with --format inputs and joined to captured traffic by input hash:

   taskdistill capture --task tickets --import gold.jsonl --format inputs
   

3. Curate, train and evaluate. Curate prints a projected cost from a 50-request sample before labelling anything and asks for --yes above $0.50; --max-usd caps the run.

   export TASKDISTILL_TEACHER_API_KEY=...    # or OPENROUTER_API_KEY
   taskdistill curate --task tickets --max-usd 2
   taskdistill train --task tickets
   taskdistill eval --task tickets --select
   

   Without gold labels, eval reports agreement with the teacher, and the default cascade target (97% agreement) needs no gold at all.

4. Serve the cascade and point your client at it. The model string your application sends can stay as it is.

   taskdistill serve --task tickets
   

   client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="unused")
   

5. Watch it. taskdistill report --task tickets --from-serve-log --since 24h recomputes the figures from real served traffic, including the observed escalation rate; a rate far above the validation estimate means the traffic has drifted away from the training data.

Results

Numbers below come from scripts/reproduce.sh (full profile, recorded teacher outputs) on 2026-09-27 on a MacBook Air, Apple M5, 24 GB (Mac17,4), macOS 26.6.2, plus one extra zero-shot evaluation (the invoices table also shows the 1.5B zero-shot base); teacher outputs were recorded on 2026-09-26. The test splits were scored 8 times (Banking77) and 9 times (invoices), each time by an evaluation shown in these tables; no choice used them: every choice (base model, seed, threshold, isotonic calibration) was made on the validation split alone. Figures marked “recorded” below (the teacher and cascade rows) replay the teacher outputs captured then, not a live call. The live bench and latency numbers are dated measurements and are not expected to reproduce exactly on different hardware or under different load. Unless noted otherwise, every score has a 95% paired bootstrap interval in brackets (1,000 resamples).

Two machines were used. Quality, calibration and training numbers come from the MacBook Air, Apple M5, 24 GB that trained the students; it is fanless and shared with other work. Cost, latency, the live bench and the cross-machine reproduction come from a Mac Studio, Apple M4 Max, 128 GB, used only for those runs. Each table names its machine.

Bottom line. Each cascade's threshold was chosen on validation; on test:

• Banking77. Target held on test: yes — Agreement against the teacher was 97.6% on test (target 97.0%) at a 23.7% escalation rate, close to the 97.1% on validation at 23.4%.
• Invoices. Target held on test: no — Agreement against the teacher met the 97.0% target on validation (98.5% at 0.0% escalation) but fell to 94.1% on test (1.7% escalation).

Metrics: agreement is how often a system gives the teacher's answer (for invoices, field micro-F1 against the teacher's JSON); accuracy and macro-F1 (F1 averaged over the 77 intents, each counted equally) are against the gold labels; ECE (expected calibration error, lower is better) is the average gap between the student's stated confidence and how often it is right; AUROC (0.5 = chance, 1.0 = perfect) is how well that confidence separates right answers from wrong ones. For invoices, JSON validity is the share of outputs that parse as a JSON object, field micro-F1 and field EM (exact match) score the 8 fields against the gold, and Doc EM is the share of documents with all 8 fields right.

Banking77 (77 intents)

Test split, n = 3,075. Cells are the score and its 95% paired bootstrap interval; a row aggregating more than one seed shows mean ± sample standard deviation only (see the per-seed table below for each seed's own score, and interval where the eval recorded one). ECE and AUROC use raw (pre-isotonic) student confidence.

Per-seed scores for student qwen2.5-0.5b (teacher labels, 3 seeds) (selected run: qwen2.5-0.5b-full-s13):

Target held on test: yes — Agreement against the teacher was 97.6% on test (target 97.0%) at a 23.7% escalation rate, close to the 97.1% on validation at 23.4%.

Invoices (8-field JSON extraction)

Test split, n = 600. The test set is 6 layouts never seen in training, so the effective sample is 6 layouts; the template-cluster bootstrap below says how wide that makes the uncertainty. Cells are the score and its 95% paired bootstrap interval (document-level). A row aggregating more than one seed shows mean ± sample standard deviation only (see the per-seed table below for each seed's own score, and interval where the eval recorded one).

Per-seed scores for student qwen2.5-0.5b (teacher labels, 3 seeds) (selected run: qwen2.5-0.5b-full-s13):

Template-cluster bootstrap over 6 groups (one per layout): 95% intervals.

Target held on test: no — Agreement against the teacher met the 97.0% target on validation (98.5% at 0.0% escalation) but fell to 94.1% on test (1.7% escalation).

Cost and latency

Energy assumptions: local cost = 20 W x wall time x $0.3/kWh; hardware amortisation off.

Banking77

Measured on a Mac Studio, Apple M4 Max, 128 GB (Mac16,9), macOS 26.5.2, 2026-09-27:

For comparison, the MacBook Air's student latency was 44.5 ms p50 / 68.2 ms p95 (in-process eval, not through the server).

Break-even at 17,419 requests at the recorded teacher cost, 1,762 requests at the list price without prompt caching.

Invoices

Measured on a Mac Studio, Apple M4 Max, 128 GB (Mac16,9), macOS 26.5.2, 2026-09-27:

For comparison, the MacBook Air's student latency was 1,391 ms p50 / 1,988 ms p95 (in-process eval, not through the server).

Break-even at 3,140 requests at the recorded teacher cost, 2,505 requests at the list price without prompt caching.

The Why section above states the teacher answered in about 594 ms at p50 and 1,242 ms at p95 (Banking77 labelling calls, 2026-09-26); that is this same recorded Banking77 teacher latency.

Live bench cross-check

Banking77 (measured on a Mac Studio, Apple M4 Max, 128 GB (Mac16,9), macOS 26.5.2, 2026-09-27)

Composed (from the test split) vs measured: p50 21.5 vs 31.3 ms, p95 854 vs 815 ms.

Invoices (measured on a Mac Studio, Apple M4 Max, 128 GB (Mac16,9), macOS 26.5.2, 2026-09-27)

Composed (from the test split) vs measured: p50 442 vs 445 ms, p95 499 vs 505 ms.

Training on the MacBook Air

Banking77

Invoices

Load average recorded alongside these runs ranged up to 4.27. The MacBook Air is fanless and can throttle under sustained load; that is why the load average is recorded next to every timing rather than assumed away.

Reproducibility on a second machine

scripts/reproduce.sh (full profile) was run again on a second machine and compared with scripts/compare_reports.py; tolerance 2.0 points on each test-split metric.

Reference: MacBook Air, Apple M5, 24 GB (Mac17,4), macOS 26.6.2, 2026-09-27. Rerun: Mac Studio, Apple M4 Max, 128 GB (Mac16,9), macOS 26.5.2, 2026-09-27.

Banking77: the selected run differed — qwen2.5-0.5b-full-s13 on the reference machine vs qwen2.5-1.5b-full-s13 on the rerun (reference: “large gains 0.97 points, below the 1.00-point minimum”; rerun: “large gains 1.26 points (>= 1.00) at 1.87x the small p95 (< 3x)”). Invoices: the selected run matched on both machines (qwen2.5-1.5b-full-s13).

All rows within tolerance across every task: no.

Outside the tolerance: Invoices student qwen2.5-0.5b (teacher labels, 3 seeds), Doc EM: 39.2% vs 41.6% (+2.33 points); Invoices student qwen2.5-1.5b (teacher labels), Doc EM: 67.5% vs 74.3% (+6.83 points); Invoices student qwen2.5-1.5b (gold labels), JSON validity: 95.0% vs 98.2% (+3.17 points); Invoices student qwen2.5-1.5b (gold labels), Field micro-F1: 93.8% vs 91.5% (-2.37 points); Invoices student qwen2.5-1.5b (gold labels), Doc EM: 78.2% vs 68.0% (-10.17 points); Invoices student qwen2.5-1.5b (gold labels), Agreement: 93.5% vs 91.0% (-2.43 points).

Calibration

Two confidence definitions were compared on validation, before either was used at test time: the primary is the trie-constrained greedy label's own renormalised token-probability product; the alternative is free greedy generation scored by the mean per-token log-probability. AUROC is against the teacher and the gold label; ECE against each.

Banking77 student qwen2.5-0.5b-full-s13, ECE on test vs gold: 15.0% raw vs 3.7% after isotonic calibration.

Invoices student qwen2.5-1.5b-full-s13, ECE on test vs gold: 14.1% raw vs 15.6% after isotonic calibration.

Choosing the base model

Banking77

Rule: use the larger base only if it gains at least 1 point on the validation metric and its p95 latency stays under 3.0x the smaller model's. Here the gain is below the 1 point minimum (+0.97 points) and the p95 ratio (2.18x) is under the 3.0x limit, so the smaller base was kept.

Invoices

Rule: use the larger base only if it gains at least 1 point on the validation metric and its p95 latency stays under 3.0x the smaller model's. Here the gain meets the 1 point minimum (+3.79 points) and the p95 ratio (1.79x) is under the 3.0x limit, so the larger base was selected.

The rule compares point estimates on the validation split (the best seed of each base); no interval is computed for this decision, so a gain close to the minimum can go either way on a rerun (see Reproducibility above).

What didn't work

• Learning-rate schedule at the spec's peak rate (quick profile, 200 iterations, 3 seeds, Banking77): on the MacBook Air (Apple M5) a constant rate reached 32.1% ± 31.8% mean validation agreement with the teacher and linear warm-up + cosine decay 68.9% ± 8.3%, with 1 and 0 of 3 seeds diverging (agreement below 10%); on the Mac Studio (Apple M4 Max) a constant rate reached 23.3% ± 38.0% mean validation agreement with the teacher and linear warm-up + cosine decay 33.9% ± 32.9%, with 2 and 1 of 3 seeds diverging (agreement below 10%). Warm-up + cosine is the default and did better on average, but it does not make short runs at this peak rate reliable. In the full profile, Banking77 converged on every seed; on invoices, the best validation checkpoint of 1 of 3 on the MacBook Air (Apple M5) and 1 of 3 on the Mac Studio (Apple M4 Max) 0.5B seeds came at or before the end of warm-up, so those students stopped early.

• The bigger Banking77 student (1.5B vs 0.5B, teacher labels) gained +0.97 points on validation agreement, below the 1 point minimum the base-model rule requires, for 2.18x the p95 latency (limit 3.0x); the rule kept the 0.5B base.

• A Banking77 student trained on the gold labels reached 92.0% accuracy against gold, above both the same student trained on teacher labels (75.5%) and the teacher itself (75.8%); part of that gap is Banking77's own label noise (Ying and Thomas, 2022, flag about 14% of the training utterances as potential label errors), which caps how high any model's accuracy against gold can go.

• Teacher prompt variants (same 200 Banking77 validation queries, same teacher model): snake-case labels (+0.5 pts accuracy at 1.3x the prompt tokens, 3.3x the cost per 1k); labels with examples (+4.0 pts accuracy at 4.3x the prompt tokens, 2.9x the cost per 1k). The spec prompt was kept for the recorded run.

• Two confidence definitions were compared on validation, before either was used at test time: Banking77 AUROC 0.877 vs 0.874 (about equal), ECE 2.8% vs 9.2%, about 3.3x lower for the primary; Invoices AUROC 0.904 vs 0.903 (about equal), ECE 3.7% vs 10.0%, about 2.7x lower for the primary; so the primary (trie-constrained token-probability product) was kept over the alternative (free greedy generation, mean per-token log-probability). Reported honestly, on test: Banking77 AUROC 0.903 vs 0.901 (about equal), ECE 3.0% vs 9.3% (lower for the primary); Invoices AUROC 0.846 vs 0.858 (higher for the alternative), ECE 14.1% vs 32.1% (lower for the primary).

• Invoices: the threshold chosen on the validation layouts did not transfer to the unseen test layouts — Agreement against the teacher met the 97.0% target on validation (98.5%, 0.0% escalation) but fell to 94.1% on test (1.7% escalation).

Spend and downloads

Total spend for the whole build: $0.251 of a $14.00 global cap (17,302 teacher calls).

Model downloads: 1.17 GB, within the 1.5 GB budget (student base 0.5B: 0.29 GB; student base 1.5B: 0.88 GB).

Configuration reference

A task lives in tasks/<task>/task.yaml next to its teacher prompt and its labels file or JSON Schema. String values can use ${NAME} or ${NAME:-default} environment references. Unknown keys are errors, and every validation error names the offending key.

Environment (see .env.example): TASKDISTILL_TEACHER_BASE_URL, TASKDISTILL_TEACHER_API_KEY (falls back to OPENROUTER_API_KEY), TASKDISTILL_TEACHER_MODEL, TASKDISTILL_BUDGET_USD (global spend cap, default 5.00), TASKDISTILL_SERVER_TOKEN (required to bind capture or serve beyond localhost) and TASKDISTILL_HOME (the workspace: store, response cache, ledger, curated data and runs; default ./.taskdistill).

Spend control. Every command that can call the teacher takes --max-usd (a cap for that run) and --yes. Before each call the ledger reserves the worst case (UTF-8 bytes of the messages plus 16 per message as prompt tokens, plus max_tokens of completion) and settles it with the usage.cost the API returns, or with the token counts times the pricing snapshot when the response has no cost, so concurrent calls can never cross a cap. Batches first run a 50-request sample and need --yes when the projection exceeds $0.50. taskdistill budget prints the spend by task and phase.

Limitations

• Platforms. The MLX path needs Apple Silicon and macOS 14 or later (the minimum of the pinned mlx wheels). On other machines train --backend mlx stops with a message pointing to --backend torch. The torch path (transformers + PEFT, Unsloth when importable with CUDA) is tested on CPU with a tiny model in CI; it has not been run on a GPU, and the Unsloth branch is covered only by a dispatch test with a mock.
• PII scrub is best-effort. It is a regex scrub with checksums for e-mail addresses, phone numbers, IBANs, card numbers, IPv4 addresses and US SSNs. It does not find names, street addresses or anything else, and it is not an anonymisation guarantee. It also creates a train/serve skew: the student trains on placeholders such as <EMAIL> but sees raw text when it serves.
• Training on Metal is not bit-for-bit deterministic. Two runs with the same seed can differ slightly; the results report the spread over seeds and the tolerance used when checking that README numbers reproduce.
• Scope. Single-turn classification and JSON extraction only: no summaries or chat replies, no multi-turn or tool-calling tasks, no multi-label classification. Streamed responses are forwarded but not captured. The server generates one request at a time (no batching) and serves one task per process. Escalations for unsupported or unparsable requests are returned verbatim; only low-confidence escalations are normalised.
• Evaluation limits. The invoice test set is 600 documents from 6 layouts never seen in training, so its effective sample is 6 layouts; the template-cluster bootstrap intervals say how wide that makes the uncertainty. The invoices are synthetic and cleaner than real mail. Banking77's gold labels are noisy (Ying and Thomas, 2022, flag about 14% of the training utterances as potential label errors), which caps any model's measured accuracy against gold.
• Cost figures are estimates where they are not measured. Local cost is an energy estimate (20 W at $0.30/kWh by default, both configurable), not a power measurement. With an inexpensive teacher whose provider caches the shared prompt, the money saved per request is small; the main gains are latency and keeping inputs on the machine.
• Latency numbers are dated measurements from two machines, and each table names its machine. Training and the in-process student latency ran on a fanless MacBook Air (Apple M5, 24 GB) that throttles under sustained load and is shared with other work; the cost and latency tables, the live bench and the break-even volumes come from a Mac Studio (Apple M4 Max, 128 GB) used only for those runs. The load average is recorded next to every timing, and live numbers are not expected to reproduce exactly on other hardware or under other load.
• Adapters are merged into the 4-bit base in memory for evaluation and serving, which avoids the extra LoRA matrix multiplications an unmerged adapter runs at every decoding step. The re-quantisation shifts confidences slightly, which is why the threshold is chosen on the same merged model that serves.
• Teacher terms. Some commercial APIs forbid using their outputs to train other models. Check your provider's terms before distilling its outputs; see ADR 5 for the checks made here.

Roadmap

• Shadow mode and canary routing for the cascade, so a new student can be compared on live traffic before it answers.
• Scheduled retraining and active learning from escalated requests (the served log already records them).
• Drift alerts on the observed escalation rate (today report --from-serve-log shows drift; nothing alerts).
• Capture of streamed responses, multi-turn and tool-calling tasks.
• Constrained JSON decoding for extraction, and request batching in the server.

Data and licences

• taskdistill is Apache-2.0 (Copyright 2026 Andrii Boiko). The synthetic invoice generator and its output are part of the repository and share that licence. A 20-document sample of the generated invoices (one per training layout, covering every trait, with their gold fields) is in examples/invoices/; scripts/make_examples.py regenerates it.

• Banking77 (Casanueva et al., 2020) is licensed under CC BY 4.0. The demo downloads the CSVs at runtime from a pinned commit of PolyAI-LDN/task-specific-datasets and checks their SHA-256. If GitHub is unavailable it falls back to the Parquet mirror legacy-datasets/banking77 on the Hugging Face Hub (install the hub extra); PolyAI/banking77 itself is a script-based Hub repository with no Parquet conversion. This repository does not redistribute the CSVs: it ships only the teacher's outputs keyed by request hash. Modifications: the official training set is split 90/10 into train and validation (stratified, seed 13), and duplicates are removed as described above.

  @inproceedings{casanueva-etal-2020-efficient,
      title = "Efficient Intent Detection with Dual Sentence Encoders",
      author = "Casanueva, I{\~n}igo and Tem{\v{c}}inas, Tadas and Gerz, Daniela and Henderson, Matthew and Vuli{\'c}, Ivan",
      booktitle = "Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI",
      year = "2020",
      publisher = "Association for Computational Linguistics",
      url = "https://aclanthology.org/2020.nlp4convai-1.5/",
      doi = "10.18653/v1/2020.nlp4convai-1.5",
      pages = "38--45"
  }
  

• Student bases: Qwen2.5-0.5B-Instruct and Qwen2.5-1.5B-Instruct are Apache-2.0; the demos use the 4-bit MLX conversions mlx-community/Qwen2.5-0.5B-Instruct-4bit and mlx-community/Qwen2.5-1.5B-Instruct-4bit at pinned revisions. No trained adapters are published.

• Teacher outputs: DeepSeek V4.1 Flash (Banking77) and DeepSeek V4 Flash 0731 (invoices) through OpenRouter, pinned to the DeepInfra provider. The DeepSeek V4 weights are MIT-licensed, DeepSeek's platform terms allow training other models on outputs, and DeepInfra's terms do not restrict the use of outputs (ADR 5). The outputs were recorded on 2026-09-26.

Licence

Apache-2.0. See LICENSE. Copyright 2026 Andrii Boiko.