shad0w

Your app asks an LLM the same kinds of small questions all day.
shad0w learns its answers and serves them in microseconds, with a written guarantee on how often it disagrees.

Website · Try it in your browser · Docs

Quick start

pip install "shad0wllm[compile]"

import shad0w

intent = shad0w.decision(
    "intent",
    options={"refund": "wants money back", "lost_card": "card lost or stolen", "balance": "asks about balance"},
    llm="openai/gpt-4o-mini",   # or anthropic/…, gemini/…, groq/…, ollama/llama3.1, any OpenAI-compatible server
)

intent("my card was stolen")    # day one: your LLM answers (~500 ms) and shad0w writes the answer down
intent.train()                  # after ~1,000 answers: builds and certifies a small table, in about a minute or less
intent("my card was stolen")    # from then on: answered locally when certified
# Decision(answer='lost_card', source='table', confidence=0.99, certified=True, flag=None, latency_us=9.6)

That is the whole integration. Anything the table is unsure about still goes to your LLM and gets logged, so the table keeps learning.

Prefer to change no code? Run the proxy and point your OpenAI client at it:

shad0w proxy --upstream https://api.openai.com/v1        # dashboard at http://localhost:8010

client = OpenAI(base_url="http://localhost:8010/v1")      # the only change
client.chat.completions.create(model="gpt-4o-mini", messages=msgs,
                               extra_headers={"X-Shad0w-Question": "intent"})   # mark which calls are decisions

Unmarked calls (chat, embeddings, streaming) pass straight through.

Is it for me?

Yes, if your app calls an LLM to pick one answer from a fixed list. For example:

What changes when you add it:

Not a fit: open-ended generation, questions that need reasoning or world knowledge, and labels that depend on tone or sarcasm (sentiment tops out near 79%; shad0w then certifies almost nothing and passes everything to your LLM, which is the correct behaviour).

How it works

1. Watch. Your LLM keeps answering. Each (text, answer) pair is written to a log.
2. Train. The log becomes a hashed word/character table with 8-bit weights. There is no GPU, no tokenizer and no human labels, and it takes seconds to a minute on a laptop.
3. Certify. On answers the table never trained on, a Learn-then-Test procedure picks a confidence threshold. Above it, the table disagrees with your LLM at most α of the time, with 90% confidence.
4. Serve. Above the threshold the table answers in microseconds. Below it, your LLM answers and the answer is logged. 1% of table answers are spot-checked against the LLM, so you can see live agreement, not only the certificate.

Use it from anywhere

Providers built in: openai, anthropic, gemini, groq, together, openrouter, mistral, deepseek, fireworks, xai, azure, ollama, vllm and lmstudio. Keys come from the usual environment variables (OPENAI_API_KEY, …). Any other OpenAI-compatible server works with base_url=. Recipes for each are in the docs.

See what it is doing

shad0w try --bundle shad0w/intent/bundle "my card was stolen" "is it raining?"
#   ✓ table     lost_card  (confidence 1.000, needs 0.911)  14.1 µs
#   → your LLM  balance    (confidence 0.717, needs 0.911)  table not sure enough: asked your model
shad0w stats --log shad0w/intent/log.jsonl     # what has been logged, and whether it is ready to train
shad0w watch                                   # live counters from a running proxy or server

• Dashboard. shad0w proxy and shad0w serve serve a live page at /. It shows the share answered by the table, LLM calls and time saved (and money, with --cost-per-call), table vs LLM latency, why the table deferred, and the live spot-check bound against α.
• Prometheus at /metrics. JSON at /v1/stats.
• Logs: SHAD0W_LOG=debug prints one line per decision.
• Hooks: on_decision=fn for Langfuse, Datadog or your own analytics; trace="decisions.jsonl" records every decision; SHAD0W_OTEL=1 emits OpenTelemetry spans.

Numbers

Measured on public datasets, with the certificate at α = 5%:

• 45–71% of LLM traffic answered by the table on 7 intent and topic tasks, with 3.9–5.7% realised disagreement. End-to-end accuracy stayed within 1 point of the LLM alone.
• ~1 µs per decision in C, 7–11 µs in JavaScript, ~10–15 µs in Python. With a 0.8B LLM teacher, mean latency fell from 75 ms to 22 ms.
• The share grows with traffic: 31% → 70% as logged answers went from 2k to 100k (AG News).
• Reproduce it on a laptop with a free teacher model: python examples/shadow_demo.py gives 69.4% answered locally on BANKING77, 4.6% realised disagreement.

How it compares

This is not new science. The certificate is Learn-then-Test, in the spirit of Trust or Escalate and BARGAIN, and the table is fastText-shaped on purpose. What shad0w adds is the packaging: one line from your existing LLM call to a certified, megabyte-sized table, with the same answers in Python, JavaScript and C.

Good to know

• The guarantee is against your LLM, not the truth. If your LLM is wrong, the table is wrong the same way. For a bound against the truth, calibrate on ~300 human labels (shad0w calibrate).
• It holds for traffic like the traffic it was certified on. Re-train when your traffic changes. The drift guard and live spot checks show you when that is.
• Runtime installs only numpy; the C core is inside the wheel. Training ([compile]) adds scipy, scikit-learn and torch. For a smaller download, install CPU-only torch first (pip install torch --index-url https://download.pytorch.org/whl/cpu); without torch, training falls back to scipy, which is slower.
• Nothing leaves your machine except the LLM calls you already make.

Contributing

Try it on one of your decisions and share your numbers, failures included. That is the most useful contribution. For code, see CONTRIBUTING.md.

@software{chauhan2026shad0w,
  author = {Chauhan, Gaurav},
  title  = {shad0w: certified microsecond decisions compiled from the model you already run},
  year   = {2026},
  url    = {https://github.com/2796gaurav/shad0w}
}

Built by Gaurav Chauhan · Apache-2.0