ThinkLess

Stop using an LLM for every decision.

ThinkLess is an open-source decision plane for AI agents. The routine judgments an agent makes (what the user wants, whether they asked for a person, which order they mean, whether a message is trying to manipulate the system) are answered by rules and small calibrated models in milliseconds. The LLM is kept for the steps that need it: writing, planning, and the decisions the small models are not sure about. Every step is traced with the plane that handled it, its confidence, latency, tokens and cost.

It is model-neutral. Rules, GLiNER 2.5, Laya, TypeSafe's Jev (and any server speaking its System One API, such as Kev and OpenJev), any Hugging Face classifier, and any LLM (local through Transformers, Ollama or vLLM, or hosted through OpenRouter, Anthropic or OpenAI) plug into the same cascade. The whole stack also runs offline on a laptop GPU, with no API key.

Why

A typical agent sends every branch of its workflow to one generative model: classify the request, extract the arguments, pick the tool, check the result, check the reply, then write the reply. Most of those are bounded questions with a handful of possible answers. A generative model answers them slowly, at full token cost, and without telling you how sure it is.

ThinkLess treats each of them as a typed question. It asks the cheapest provider first and escalates only when the answer's confidence is below a threshold you set per question and per provider, from data. Your code reads a typed decision and never needs to know which model produced it.

intent = engine.decide(ticket, INTENT)          # rules, then GLiNER, then Laya, then the LLM
if intent.is_("refund_duplicate_charge"):       # only true when the answer is confident
    ...

Results

Same agent, same 53 labeled support tickets, two decision planes: llm sends every decision to the LLM, hybrid asks rules, GLiNER and Laya first. Hosted models ran through OpenRouter, and the dollar figures are what OpenRouter billed (full results):

• Hybrid matched or beat the LLM-only design on every model, at 40 to 44 percent lower billed cost and 35 to 55 percent less time spent on decisions. 87 percent of hybrid decisions never reached the LLM.
• Small models are the most reliable extractors. Rules and GLiNER found every order number on every run. The larger hosted LLMs did too; Qwen 3.7 Flash and the local 1.7B model missed ones written without the word "order".
• On Banking77 (77 intents, 500 examples) the cascade matched or beat its LLM at a quarter of the cost. GLiNER alone 71.8 percent; Qwen 3.7 Flash alone 73.8 percent, cascade 74.8 percent; Claude Haiku 4.5 alone 76.2 percent, cascade 76.2 percent. Each time a quarter of the calls reached the LLM.
• Nothing was tuned on the test tickets. Thresholds and every fix were validated on a separate calibration set first. The baseline is the strongest form of the LLM design: all six triage questions in one prompt.
• The support set is 53 synthetic tickets: evidence of the mechanism and its failure modes, not a leaderboard. The benchmarks page lists the weak areas found and what was done about each.

Quick start

pip install "thinkless[local]"        # GLiNER, Laya and a local LLM
thinkless doctor                      # check the environment

from thinkless import Choice, Engine, Extract, JSONLSink, Tracer, YesNo
from thinkless.llm import TransformersLLM
from thinkless.providers import GLiNER, Laya, LLMDecider, Rules

rules = Rules()
rules.match("wants_human", r"\b(real person|a human|an agent|operator)\b", True, field="message")
rules.extract("order", field="message", order_id=r"#\s?(\d{4,5})\b")

llm = TransformersLLM("Qwen/Qwen3-1.7B")
engine = Engine(
    [rules, GLiNER(), Laya(), LLMDecider(llm)],     # cheapest first
    llm=llm,
    threshold=0.8,
    tracer=Tracer([JSONLSink(".thinkless/traces")]),
)

with engine.run("ticket") as run:
    decisions = engine.decide_many(
        {"message": "I was charged twice for order #4471. Refund it or I'm cancelling."},
        [
            Choice("What does the customer want?", name="intent",
                   options={"refund": "wants money back", "order_status": "asks where an order is",
                            "cancel": "wants to cancel", "other": "anything else"}),
            YesNo("Is the customer asking for a person?", name="wants_human"),
            Extract(name="order", fields={"order_id": "the order number"}),
        ],
    )

for name, d in decisions.items():
    print(f"{name:12} {d.value!s:24} {d.status.value:9} via {d.provider}")
print(run.summary().decisions_by_plane)      # which plane answered what

Prefer a hosted model? Put OPENROUTER_API_KEY=... in .env and swap one line; OpenRouter reaches Qwen, Gemini, GPT and Claude models with one key:

from thinkless.llm import OpenRouterLLM

llm = OpenRouterLLM("qwen/qwen3.7-flash", reasoning={"enabled": False})

Then open the trace:

thinkless trace view

No GPU? pip install thinkless and run examples/01_rules_only.py: the API, the cascade and the traces with no model downloads.

Use it in your agent

ThinkLess sits inside the framework you already use, at the points where the agent decides something: which path to take, whether an input is safe, whether a tool call is inside policy. Uncertain decisions go to the code you run today, so nothing gets worse while the routine ones get cheaper.

from thinkless.integrations.langgraph import router

intent = router(engine, INTENT, {"order_status": "tracking", "refund": "refunds"},
                default="agent")          # your existing LLM node
builder.add_conditional_edges(START, intent, intent.destinations)

• LangGraph: routers, decision nodes and gated tools.
• OpenAI Agents SDK: input guardrails, tool guardrails and routing to a specialist agent.
• Any framework: route and gate for Pydantic AI, a hand-written loop, or anything else.

Before switching anything, measure it. Shadow mode runs ThinkLess next to your current code on live traffic and reports, per decision, the agreement with a confidence interval and what it would save:

from thinkless.shadow import Shadow

shadow = Shadow(engine, log="shadow/intent.jsonl", sample=0.2)

@shadow.watch(INTENT, cost_usd=0.0004)    # today's cost per call
def classify(message: str) -> str:
    ...                                    # unchanged, and still what users get

thinkless shadow report shadow/intent.jsonl --volume 2000000

The migration guide walks through it, and the FAQ covers the common questions. To share one engine across services, run it as a decision server or MCP tools.

Try the demo agent

A complete customer support agent ships with the package: a mock store with orders, payments and a help center, 53 labeled tickets, and every hard case we could think of (order numbers written in ways a regex misses, a customer asking about someone else's order, prompt injections, requests for a human, Spanish and German tickets).

thinkless demo --ticket T-016                          # one ticket, printed as a trace tree
thinkless demo --ticket T-051 --mode hybrid --mode llm # compare decision planes
thinkless bench support                                # every ticket in every mode
thinkless bench support --llm openrouter:qwen/qwen3.7-flash --reasoning off
thinkless bench intents --dataset banking77

How it works

Typed questions. Choice, Score, YesNo and Extract declare what you want to know and which answers are allowed. They follow the System One wire format, so the same question works on every provider. Questions

A confidence cascade. Providers are tried cheapest first. Each gets one call with every open question it supports; answers that clear their threshold close, the rest move on. Unresolved questions come back uncertain with the best answer seen, and decision.is_() never matches an uncertain answer. The cascade

One meaning of confidence. Providers disagree about what "confidence" means (Laya and TypeSafe compute it differently), so a threshold of 0.8 would mean different things per backend. ThinkLess derives a single normalized confidence from each provider's probabilities. Confidence

Thresholds from data. thinkless calibrate sweeps thresholds on labeled examples and recommends the lowest one that meets your accuracy target, per question and per provider (thresholds={"intent@gliner": 0.9}). Calibration

Traces for everything. Decisions, provider attempts, generations, tool calls and rule checks are spans with W3C-compatible ids. Write JSONL audit files, export to any OpenTelemetry backend with GenAI semantic conventions, or open the self-contained HTML viewer. Content capture can be switched off without losing structure, timings or confidences. Tracing

Providers and backends

Writing your own provider is one class. Providers, LLM backends

Documentation

• Getting started, installation
• Use it in your agent: overview, LangGraph, OpenAI Agents SDK, any framework, migration, shadow mode, FAQ
• Concepts: the four planes, questions, confidence, the cascade, tracing
• Guides: the support agent, calibration, providers, LLM backends, production, serving, running a pilot, troubleshooting
• Benchmarks, CLI reference, Python API, roadmap

Status

ThinkLess is pre-1.0: the core API (questions, the engine, decisions and traces) is meant to stay stable, and providers, adapters and benchmarks will grow. The decision benchmark compares decision models on eight public tasks and is open to submissions. Next up are more models on it, cross-request batching in the server, and calibrated LLM confidence from log probabilities. See the roadmap.

Contributing

Issues, providers, benchmark runs on other hardware and models, and new demo scenarios are all welcome. Start with CONTRIBUTING.md. The bar for changes that affect accuracy, latency or cost is evidence: a before and after from thinkless bench or thinkless calibrate.

Sponsoring

If ThinkLess saves your team time or money, consider sponsoring its development.

Model vendors can sponsor runs of their models on the decision benchmark. Sponsored runs are done by the maintainers and published as verified, whatever the result: sponsorship buys runs, not rankings. Sponsorship also funds more providers and more demos.

Citing

See CITATION.cff.

Acknowledgements

ThinkLess builds on the work of the teams behind Jev and the System One API (TypeSafe AI), GLiNER2 (Fastino), Laya (Convai Innovations), Qwen and Banking77 (PolyAI).

ThinkLess is not affiliated with the NeurIPS 2025 paper Thinkless: LLM Learns When to Think, which studies adaptive reasoning inside a single model.

License

Apache License 2.0.