Skip to main content

sureshot

PyPI Python CI License

Fast, calibrated answers from a cheap model, with escalation to a stronger model only when the cheap one isn't sure.

You ask structured questions about a piece of content (an email, a ticket, an image). System 1 is a fast model that gives a probability for each option: a small LLM with logprobs, or a classifier trained on your own labels. System 2 is a stronger model that answers with a reason. The router asks System 1 first and sends only the questions it is unsure about to System 2, so most decisions cost one cheap call.

content + questions ─▶ System 1 ─▶ confident? ── yes ──▶ answer (source: sureshot)
                                      │
                                      no ──▶ System 2 ──▶ answer + reason (source: system2)

As you label answers, sureshot learns how far to trust System 1 on each question, trains a cheaper tier, and warns you when things change.

Install

pip install sureshot                # any OpenAI-compatible or HTTP model, as System 1 or System 2
pip install "sureshot[claude]"      # adds Claude as System 2
pip install "sureshot[api]"         # adds the HTTP API

Every other extra (Postgres, the local classifier, MCP, Parquet, OpenTelemetry, Argilla) is listed in the guide.

Quick start

With OPENAI_API_KEY and ANTHROPIC_API_KEY set:

from sureshot import Router, OpenAIDecider, ClaudeAPI

questions = {
    "category": {"type": "choice", "instructions": "What kind of email is this?",
                 "criteria": {"work": "From a real person", "promo": "Marketing"}},
    "needs_reply": {"type": "noul", "instructions": "Does this email need a personal reply?"},
}

router = Router(OpenAIDecider("gpt-4o-mini"), ClaudeAPI(), threshold=0.8)
result = router.decide(email_text, questions)

result["answers"]["category"]   # {"answer": "work", "confidence": 0.97, "source": "sureshot", ...}
result["escalated"]             # the questions System 2 answered, e.g. ["needs_reply"]

A question is a choice (one of named options), a score (a level from an ordered list) or a noul (yes or no). Any model works on either side: Mistral, DeepSeek, Together, OpenRouter, Ollama, vLLM and others have built-in classes, and anything else plugs in over HTTP (models). System 1 must return logprobs, so .check() a decider before using it.

Learn thresholds from your labels

A fixed threshold of 0.8 is only a starting point: a confidence of 0.8 rarely means 80% right. Log decisions, label some of them, and let sureshot find the threshold that meets your accuracy target:

from sureshot import Store, calibrate, review_queue

store = Store("sureshot.db")
router = Router(OpenAIDecider("gpt-4o-mini"), ClaudeAPI(), log=store, audit=0.05)

for item in review_queue(store, limit=30, sample=0.02, confident=0.1):   # the answers worth labeling
    store.label(item["id"], item["key"], true_answer)

calibrate(store, target_accuracy=0.95, sample=0.02, confident=0.1)       # the router uses these thresholds

The feedback loop covers the rest: training a local classifier as a cheaper first tier, rechecking thresholds, and spotting drift.

HTTP API and configuration

Run sureshot as a service, configured from environment variables:

docker run -p 8000:8000 -v sureshot-data:/data \
  -e SURESHOT_API_KEY=... -e SURESHOT_SYSTEM1_MODEL=gpt-4o-mini -e SURESHOT_SYSTEM1_PROVIDER=openai \
  -e OPENAI_API_KEY=... -e ANTHROPIC_API_KEY=... \
  ghcr.io/maxtmiller/sureshot

curl -s localhost:8000/decide -H "Authorization: Bearer $SURESHOT_API_KEY" -H "Content-Type: application/json" \
  -d '{"state": "...", "questions": {...}}'

Or pip install "sureshot[api]" and sureshot serve. The routes and settings are in the guide.

Where next

Development

pip install -e ".[dev]" && pytest

Metadata

Release files for sureshot 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sureshot 0.8.0
File Size Uploaded Provenance Actions
sureshot-0.8.0.tar.gz 163.9 kB With provenance Details for sureshot-0.8.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sureshot 0.8.0
File Interpreter ABI Platform Actions
sureshot-0.8.0-py3-none-any.whl Python 3 none any Details for sureshot-0.8.0-py3-none-any.whl

Total release size: 283.9 kB

Release files / sureshot-0.8.0.tar.gz

Download URL sureshot-0.8.0.tar.gz
Size 163.9 kB
Tags Source
SHA-256 checksum
How to use checksums
f45ed2bdc2b2c4d617c81553bc1ce57be2b0958ff84dc9e7c79063f23d1d52fe
BLAKE2b-256 checksum
How to use checksums
312547e102e7d04fb4655bfbd5a7dfae64d759e5dddfe5f88dae4c8b3561d481
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Release files / sureshot-0.8.0-py3-none-any.whl

Download URL sureshot-0.8.0-py3-none-any.whl
Size 120.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9812a35d7d3865dd837d3121e9b577a25b057fab9d0eaa2eafec8acfb4c0ebc5
BLAKE2b-256 checksum
How to use checksums
5ea2c0284bc09d9a3286585cf68fbf44e393baa453b0fdeab8e3d4b1fa7d7dfd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Security for sureshot 0.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for sureshot-0.8.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page