llm-sidecar

A local sidecar that gives every tool on your machine grounded, cited, routed AI — and never asks you which model to use.

One process. It picks a working model, searches the web, reads pages, checks facts, and tells you what everything cost. Works with no API key at all if you have Ollama; works better with one.

$ llm-sidecar answer "What is Iran's population?"
As of mid-2026, Iran's population is estimated at 93,168,497 (Worldometer,
based on UN 2024 Revision). The 2025 estimate from Wikipedia is 92,417,681.

Caveat: Sources differ by year and methodology: Worldometer (2026 mid-year)
gives 93.17 million; Wikipedia (2025 est.) gives 92.42 million.

Sources:
  · https://en.wikipedia.org/wiki/Demographics_of_Iran
  · https://www.worldometers.info/world-population/iran-population/

That came from the live web, not training data, and it says where it got it — including that the sources disagree. Ask something the web can't settle and it tells you, instead of guessing.

It is not instant: a grounded answer means a search, two or three page fetches and a model call, so expect tens of seconds — more on free models, which are slow and sometimes have to be rotated past.

Status: early (0.4.0). The API may still move. Extracted from Agora, where the routing and verification were originally built and proven.

Sixty seconds

pip install "llm-sidecar[all]"
llm-sidecar serve            # dashboard at http://localhost:4001

Or clone it, which also gets you the setup scripts:

git clone https://github.com/awaistechnologist/llm-sidecar
cd llm-sidecar
./install.sh          # Windows: install.bat
./run.sh              # Windows: run.bat

install.sh builds a virtualenv, installs everything, then tells you what it can actually reach — whether Ollama is running and how many models you have, whether a key is set, whether Docker is available for SearXNG — and prints the MCP config with your real paths filled in. Safe to re-run.

run.sh starts the daemon and opens the dashboard once it's answering. It passes arguments through (./run.sh --port 4100, ./run.sh --no-ui) and reads a .env in the project directory if you keep your key there.

Open the dashboard: a chat window, every capability in a Tools tab, and a live view of what is being chosen and what it costs. No key required.

Python 3.11+. The core needs only httpx — llm-sidecar on its own is a seven-package install. The extras add what each door needs:

pip install llm-sidecar              # library only: complete, stream
pip install "llm-sidecar[search]"    # + web search and read_url
pip install "llm-sidecar[daemon]"    # + HTTP server and dashboard
pip install "llm-sidecar[mcp]"       # + MCP server
pip install "llm-sidecar[all]"

What it does

Why it exists

Most "LLM router" projects answer which provider should this call go to. That's plumbing. The point here is the layer above it: capabilities that are hard to build well and that every tool re-implements badly — grounded search, cited verification — sharing one routing, budget and cost substrate.

The routing does earn its keep in one specific way: a catalogue entry is not a working model. Free tiers throttle, checkpoints get retired, endpoints start returning empty completions. So before handing back a model, the picker spends one tiny call proving it answers right now.

Four ways in

Same core, four doors, because the consumers can't use each other's interface.

Library

from llm_sidecar import Sidecar
sc = Sidecar()

a = sc.answer("What shipped in Python 3.14?")
if a.grounded:                          # False = the sources didn't settle it
    print(a.text, a.sources)

sc.verify(["The Eiffel Tower is in Berlin"])      # → contradicted, cited
sc.fact_check(article)                             # extract claims, verify each
sc.summarise(text, style="bullets", focus="security")
sc.classify(tickets, labels=["bug", "feature", "question"])
sc.extract(invoice, {"total": "amount with currency", "due_date": "ISO date"})

sc.complete("Explain CRDTs")                       # routes itself
sc.complete(messages=[...])                        # real multi-turn
sc.complete_many([p1, p2, p3])                     # concurrent, ordered
await sc.acomplete("...")

sc.local_models()   # your Ollama models, scored against this machine
sc.usage(days=30)   # what you spent
sc.status()         # everything at once

HTTP daemon — for any tool, in any language

llm-sidecar serve
export OPENAI_BASE_URL=http://localhost:4001/v1
export OPENAI_API_KEY=unused        # the format demands the field

It speaks the chat-completions format every provider copied from OpenAI. That format is a de facto standard, not a vendor tie — Ollama and OpenRouter accept the identical request shape, which is why routing between them is a URL swap. The variable is named after OpenAI because the openai SDK reads it; other tools call the same setting --openai-api-base, apiBase, or "Base URL".

The model field is a request, not an instruction. A tool that hardcodes gpt-4o gets a verified working model and never finds out:

$ curl -s localhost:4001/v1/chat/completions -d '{"model":"gpt-4o", ...}'
{"model": "nvidia/nemotron-3-ultra-550b-a55b:free", ...,
 "x_sidecar": {"cost_usd": 0.0, "local": false, "cached": false}}

Beyond the standard surface: /v1/answer, /v1/verify, /ops/* for each capability, /status, /usage, /resolve-preview, /config/*. Swagger at /docs.

MCP — for agents

{"mcpServers": {"llm-sidecar": {
  "command": "/path/to/venv/bin/python", "args": ["-m", "llm_sidecar.mcp_server"]}}}

answer_question · search_web · read_url · verify_claims · extract_claims · fact_check_document · summarise · classify · extract_fields · delegate · usage_report · sidecar_status

Note what's not there: a general "call an LLM" tool. An MCP client is already a model, so exposing inference to it is close to a no-op. What it can't do for itself is fetch live evidence and grade claims against it. delegate is the exception — it's for offloading bulk work to something cheap.

CLI

llm-sidecar answer "who currently runs the ECB?"   # non-zero exit if ungrounded
llm-sidecar verify "the Great Wall is visible from space"
llm-sidecar sum report.md --style bullets
llm-sidecar models                                  # local models vs your RAM
llm-sidecar usage --days 30
llm-sidecar searxng up                              # better search, one command

How a model gets chosen

Two questions, always. Tier = how capable. Budget = what it may cost. They are independent.

What each capability asks for

  verify · fact_check · summarise · classify        ┐
  extract · extract_claims · delegate               ├──► tier "fast"
                                                    ┘    bulk work, cheap

  answer · chat · complete()                        ───► tier "balanced"

  budget is always your configured default unless you pass one.

There is no separate "chat model". One picker serves everything, so changing a key or a budget changes every capability at once.

The resolution order

Four checks, first match wins.

  ① model="ollama/qwen2.5:32b" given?   ──yes──►  use it. never rotated.
                    │ no
  ② is this tier LOCKED to a model?     ──yes──►  use it. budget ignored.
                    │ no
  ③ resolved this tier+budget <15m ago? ──yes──►  reuse it.
                    │ no
  ④ resolve ▼

     budget picks the candidate pool
     ┌──────────────────────────────────────────────┐
     │ free + key     free cloud models, Ollama next│
     │ free + no key  Ollama only                   │
     │ cheap          paid, under $1 per M tokens   │
     │ best           paid, over $5 per M tokens    │
     └──────────────────────────────────────────────┘
                    │
          probe 3 at once: "Reply OK"
                    │
        ┌───────────┴────────────┐
   one answers               all 3 fail
        │                         │
  use it, cache 15m       probe the next 3 …
                                  │
                        nothing left → NoWorkingModel

Probing concurrently changes only how fast a model is found — the highest-priority success still wins, not whichever replied first. If a model passes the probe and then fails the real call, it's marked dead for the process and the next request routes elsewhere.

Which free cloud model, specifically? A curated list first, then the rest of the pool by context length. Curated entries missing from the live catalogue are skipped silently, which happens constantly — of nine curated free picks, one survived to today. So in practice: the survivors, then the roomiest.

When free OpenRouter models are used

Exactly one combination reaches them. Tier is irrelevant to free-vs-paid.

With a key and budget=free, the three tiers get three different free models — pushing parallel work through one free endpoint is how you collect 429s.

Locking a tier

Normally a tier chooses its own model and re-checks every 15 minutes. Locking overrides that: always this model, no search, no probe, no budget — a locked tier stays put even if you ask for best.

LLM_SIDECAR_MODEL_FAST=ollama/gemma3:27b          # env

dashboard → Local models → click "fast" on a row
POST /config/tier {"tier": "fast", "model": "..."}

Worth it for reproducibility and no probe latency. The cost: a locked tier never rotates away, so if that model starts failing, your calls fail with it.

auto is not a mode

model can only carry one axis:

auto is an unrecognised string, and every unrecognised string means "use both defaults" — which is exactly what lets a tool hardcoding gpt-4o work. Because one field can't say "powerful and best", the daemon also takes a separate budget field, and the dashboard has two selectors.

Not sure what you'd get? GET /resolve-preview?tier=fast&budget=free tells you — current state, the candidate order, and why — without probing anything.

What answered, and what it cost

Retrieval: who does the searching

SearXNG

llm-sidecar searxng up

Writes a compose file and settings to ~/.config/llm-sidecar/searxng/, generates a secret, starts the container, and waits until it really answers a JSON query before reporting success. Then it's detected automatically.

Why it needs a command: SearXNG ships with its JSON API disabled, so the stock image returns 403 to every request, which reads as "unavailable" and falls back silently — the failure looks like nothing happening. The shipped settings enable formats: [html, json] and turn off the bot limiter, which protects public instances and here would only throttle you. Bound to 127.0.0.1: no auth, no limiter, so exposing it would hand anyone a free search proxy on your address.

Already running one? Point SEARXNG_URL at it.

OpenRouter

Not a search API. Its web plugin retrieves inside a chat completion — you can't get results without paying for a completion too — which is why it's a mode on the answer path rather than a search provider.

sc.answer("...", via="openrouter")     # billed, ~$4 per 1000 results
sc.complete("...", web=True)

Worth it when local search is being CAPTCHA'd: retrieval happens on their side, against better sources. Never a default, never implicit, and never cached — paying for retrieval and then serving a stored answer defeats the point.

Dashboard

llm-sidecar serve also serves a dashboard at http://localhost:4001.

Dashboard — API key and budget, status, which local models fit, spend over 30 days, cache size, whether SearXNG is really being used. Chat — multi-turn, streaming, with the model and cost under every reply. Tools — every capability, in one place, to try before you wire it in.

One HTML file. No build step, no dependencies, no external requests — no CDN, no fonts, no analytics; there's a test asserting it. Settings apply to the running daemon and are not written to disk unless you tick "remember".

Opt out entirely with llm-sidecar serve --no-ui (or LLM_SIDECAR_NO_UI); / then 404s and the API is untouched. That controls serving, not installing — the page is a ~40 KB file in the package either way.

Configuration

Precedence: defaults < ~/.config/llm-sidecar/config.json < environment < keyword arguments to Sidecar(...).

The daemon binds loopback deliberately: it holds an API key and spends real money on request. config.save() does not write the API key unless asked.

Caching and cost

Deterministic requests (temperature=0) and searches are cached to disk, which is why re-checking a document doesn't re-pay for the claims that didn't change. Creative requests are deliberately not cached — a byte-identical "random" answer is a surprise, not an optimisation. The cache is trimmed oldest-first to 256 MiB; the ledger rotates at 8 MiB.

Performance

Measured on an M3 Max, not estimated.

Honest limitations

• Verification is only as good as retrieval. The verifier can only grade what came back; ambiguous evidence produces a wrong verdict. Unverified claims are re-checked against full page text, and SearXNG covers more engines — neither is a fix. When retrieval finds nothing useful you get unverified, and that is the correct answer.
• read_url strips <nav>/<header>/<footer> and finds the content region, but it's regex-based, not a readability port. Unusual markup degrades to the whole page.
• Streaming bypasses the completion cache. Replaying stored tokens is a different feature.
• The daemon has no request queue or admission control. Fine for one user, wrong for anything shared.
• Parallel verification helps against cloud models, not against a single local Ollama model — those queue server-side anyway.
• No embeddings, vector store, or cross-session memory. Different product.
• searxng drives Docker or Podman compose. Anything else: run the compose file yourself and set SEARXNG_URL.

Layout

The core imports nothing but httpx. FastAPI and mcp are optional extras, pulled in only by the doors that need them:

pip install -e ".[search]"    # web search
pip install -e ".[daemon]"    # HTTP server + dashboard
pip install -e ".[mcp]"       # MCP server
pip install -e ".[all]"

Tests

pytest

152 tests, fully offline — every network path is stubbed. Live-provider behaviour is verified by hand.

Licence

MIT.