Skip to main content

bounded-distill

Distill your LLM classifier into a small, fast model — with a statistical bound on agreement.

Status: pre-alpha (planning). Version 0.0.1 contains no working API yet. This README describes the planned design. Feedback is welcome.

The problem

Many teams classify text with an LLM pipeline: a prompt, a provider, output parsing. It works, but every call is slow and costs money. A common fix is to train a small classifier on the LLM's own labels and fall back to the LLM only when the small model is unsure.

Training that classifier is easy. Deciding when it is safe to trust it is the hard part. Nobody wants to replace an LLM in production on the basis of "it looked fine on a validation set".

What bounded-distill does

You give it texts and the labels your LLM already produced (for example, from your logs). You state a target, such as "agree with the LLM at least 97% of the time". bounded-distill:

  1. trains a small classifier (frozen sentence embeddings + logistic regression);
  2. picks a confidence threshold so that the lower confidence bound of agreement on the texts it answers meets your target — not just the point estimate;
  3. reports how much traffic the small model can take over, and the estimated cost and latency savings;
  4. lists suspects: texts where the small model is confident but disagrees with the LLM. These are often LLM labeling mistakes and point to ambiguous class definitions in your prompt.

Texts below the threshold go back to your own LLM pipeline. The library never calls an LLM itself.

Planned API

from bounded_distill import distill

router = distill(texts, llm_labels, target_agreement=0.97)

label = router.predict(text)
if label is None:              # not confident enough
    label = my_llm_pipeline(text)

print(router.report())         # coverage, agreement (with bound), per-class stats, savings
router.suspects(top_n=50)      # confident disagreements worth a human look

Optional: pass a small gold-labeled set to see accuracy next to agreement with the LLM.

How it differs from similar tools

Several projects distill an LLM classifier into a small model with an LLM fallback. bounded-distill focuses on:

  • Offline first. Works on labels you already have; it does not run or wrap the LLM.
  • A statistical bound, not a hand-picked threshold.
  • Auditing the teacher. Confident disagreements surface LLM labeling errors (in the spirit of Confident Learning, Northcutt et al., 2021).
  • Small and simple. One function call, strict typing, few dependencies.

Honest limitations

  • Agreement with the LLM is not accuracy. If the LLM makes the same mistake consistently, the small model learns it too. Use a gold set to check.
  • Single-label classification with a fixed, known set of labels only.
  • When the small model answers, you get only the label, not other fields of your pipeline's structured output.

Roadmap

  • Offline core: distill, Router.predict, save/load
  • Report: coverage, agreement with lower bound, per-class metrics, savings
  • Suspects (confident disagreements)
  • Benchmarks on public datasets with real LLM labels
  • Optional online mode (collect pairs from a running pipeline)

License

MIT

Metadata

Release files for bounded-distill 0.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for bounded-distill 0.0.1
File Size Uploaded Provenance Actions
bounded_distill-0.0.1.tar.gz 3.6 kB Without provenance Details for bounded_distill-0.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for bounded-distill 0.0.1
File Interpreter ABI Platform Actions
bounded_distill-0.0.1-py3-none-any.whl Python 3 none any Details for bounded_distill-0.0.1-py3-none-any.whl

Total release size: 7.8 kB

Release files / bounded_distill-0.0.1.tar.gz

Download URL bounded_distill-0.0.1.tar.gz
Size 3.6 kB
Tags Source
SHA-256 checksum
How to use checksums
ab8a75afffb8bdfdaf5fdb1ed09c5e80e5cb34b8ba0d2a3bfe697c7a7fd41b14
BLAKE2b-256 checksum
How to use checksums
53d1b3add4f979d100dd29bb10f4e811189eeba1997533c46e6a714da6dd062b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.3 {"installer":{"name":"uv","version":"0.11.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / bounded_distill-0.0.1-py3-none-any.whl

Download URL bounded_distill-0.0.1-py3-none-any.whl
Size 4.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
eb516a1be68f6d78536fe7621a979a3fb4506d02a6e90a6d4473af40a94d4084
BLAKE2b-256 checksum
How to use checksums
6f0ea839f6be200a27e99122183d582010067ff1def348c43bc0902d4a89ead5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.3 {"installer":{"name":"uv","version":"0.11.3","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for bounded-distill 0.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page