bounded-distill
Distill your LLM classifier into a small, fast model — with a statistical bound on agreement.
Status: pre-alpha (planning). Version 0.0.1 contains no working API yet. This README describes the planned design. Feedback is welcome.
The problem
Many teams classify text with an LLM pipeline: a prompt, a provider, output parsing. It works, but every call is slow and costs money. A common fix is to train a small classifier on the LLM's own labels and fall back to the LLM only when the small model is unsure.
Training that classifier is easy. Deciding when it is safe to trust it is the hard part. Nobody wants to replace an LLM in production on the basis of "it looked fine on a validation set".
What bounded-distill does
You give it texts and the labels your LLM already produced (for example, from your logs). You state a target, such as "agree with the LLM at least 97% of the time". bounded-distill:
- trains a small classifier (frozen sentence embeddings + logistic regression);
- picks a confidence threshold so that the lower confidence bound of agreement on the texts it answers meets your target — not just the point estimate;
- reports how much traffic the small model can take over, and the estimated cost and latency savings;
- lists suspects: texts where the small model is confident but disagrees with the LLM. These are often LLM labeling mistakes and point to ambiguous class definitions in your prompt.
Texts below the threshold go back to your own LLM pipeline. The library never calls an LLM itself.
Planned API
from bounded_distill import distill
router = distill(texts, llm_labels, target_agreement=0.97)
label = router.predict(text)
if label is None: # not confident enough
label = my_llm_pipeline(text)
print(router.report()) # coverage, agreement (with bound), per-class stats, savings
router.suspects(top_n=50) # confident disagreements worth a human look
Optional: pass a small gold-labeled set to see accuracy next to agreement with the LLM.
How it differs from similar tools
Several projects distill an LLM classifier into a small model with an LLM fallback. bounded-distill focuses on:
- Offline first. Works on labels you already have; it does not run or wrap the LLM.
- A statistical bound, not a hand-picked threshold.
- Auditing the teacher. Confident disagreements surface LLM labeling errors (in the spirit of Confident Learning, Northcutt et al., 2021).
- Small and simple. One function call, strict typing, few dependencies.
Honest limitations
- Agreement with the LLM is not accuracy. If the LLM makes the same mistake consistently, the small model learns it too. Use a gold set to check.
- Single-label classification with a fixed, known set of labels only.
- When the small model answers, you get only the label, not other fields of your pipeline's structured output.
Roadmap
- Offline core:
distill,Router.predict, save/load - Report: coverage, agreement with lower bound, per-class metrics, savings
- Suspects (confident disagreements)
- Benchmarks on public datasets with real LLM labels
- Optional online mode (collect pairs from a running pipeline)
License
MIT
Metadata
Release files for bounded-distill 0.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| bounded_distill-0.0.1.tar.gz | 3.6 kB | Without provenance | Details for bounded_distill-0.0.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| bounded_distill-0.0.1-py3-none-any.whl | Python 3 | none | any | Details for bounded_distill-0.0.1-py3-none-any.whl |
Total release size: 7.8 kB
Release files / bounded_distill-0.0.1.tar.gz
| Download URL | bounded_distill-0.0.1.tar.gz |
|---|---|
| Size | 3.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ab8a75afffb8bdfdaf5fdb1ed09c5e80e5cb34b8ba0d2a3bfe697c7a7fd41b14
|
|
BLAKE2b-256 checksum How to use checksums |
53d1b3add4f979d100dd29bb10f4e811189eeba1997533c46e6a714da6dd062b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.3 {"installer":{"name":"uv","version":"0.11.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / bounded_distill-0.0.1-py3-none-any.whl
| Download URL | bounded_distill-0.0.1-py3-none-any.whl |
|---|---|
| Size | 4.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
eb516a1be68f6d78536fe7621a979a3fb4506d02a6e90a6d4473af40a94d4084
|
|
BLAKE2b-256 checksum How to use checksums |
6f0ea839f6be200a27e99122183d582010067ff1def348c43bc0902d4a89ead5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.3 {"installer":{"name":"uv","version":"0.11.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for bounded-distill 0.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.