adk-smart-router

A predictive semantic router for Google ADK agents. It does two things: it prunes the tool menu once per session based on what the user asked for, and it skips tool calls whose answer it already has.

Both are aimed at the same problem. An agent with fifty tools sends all fifty declarations to the model on every turn, whether the user asked about a refund or the weather. The declarations cost tokens, and a long menu costs accuracy, because the model has more wrong answers to choose from.

Status

It works, and the saving is measured rather than projected.

The headline comes from a live A/B: 600 sessions and 1,894 turns of a 100-tool customer support agent, half routed and half with the router uninstalled, for $12.72. The run is real, and the conversations in it are not. Every model call, every billed token and every cache hit below was actually incurred, and the arm split was made by the shipped code rather than by the test harness, but the user turns were written by this project because there is no production traffic to use yet. Read it as a measurement on authored traffic. The plugin ships with the same holdout arm enabled, so the first number from real users comes from an adopter running report_ab.py on their own.

The two savings differ only by that turn count, which is the first direct evidence that a pruned menu does not cost an agent extra rounds. Quality did not move either. Both arms called a tool in the same share of sessions, made the same number of calls per session, reached about the same number of distinct tools, and sent the same share of calls into the long tail. A router hiding tools people needed would show up as a shorter tail, and it does not.

Read −22.5% as a floor. Raw prompt tokens fell 49.7%, and the whole gap between that and the bill is the cache. Gemini Enterprise charges a tenth as much for a cached input token, so the declarations the router removes were already nearly free: you save a tenth of what you remove, and only while the cache keeps hitting. The A/B's control arm sends one tool list, so all 289 of its sessions keep one prefix warm; the routed arm splits its sessions over 24 menus, so each menu mostly waits. That is the experiment's shape rather than the router's. Repricing every routed opening at the warmth the routed arm itself reaches when a menu is busy reads 32.1%, and closing the rest of the gap would read 42.4% as an upper bound. Denser traffic per menu is what moves the number up that range, which is why the router sends one of a few fixed menus rather than a bespoke set per session.

The worst case has been run too. An agent small enough that pruning carries its shared prefix under the model's cache floor can never open a session warm. Twelve sessions of one confirm it: turn one cached exactly zero tokens every time, against an unpruned control that opened warm seven times in eleven. It still billed 43.4% less per turn, which is why qualify warns about this case rather than refusing it.

The whole thing has been run on somebody else's data. Salesforce/xlam-function-calling-60k is a public function-calling benchmark: 60,000 request-and-answer pairs over a catalogue of real, executable third-party APIs, built by Salesforce AI Research with the APIGen pipeline. The APIs are real and so is the answer key. The requests are not user traffic. They are model-written and then verified in three stages, format, actual execution against the API, and a semantic check that the result answers the question; a human read of 600 samples found over 95% of them correct. The dataset is CC BY 4.0 and gated behind accepting that licence on Hugging Face. None of it was written here, which is the point: it is the one surface where nothing about this project shaped the data.

The slice this repository routes over holds 3,179 distinct APIs. The prediction rule reaches 0.9960 set recall at a mean menu width of 2.0 tools on a 100-tool slice of it, and 0.9666 at 5.7 tools across all 3,179, a 99.8% prune, with no retuning. Then a live 599-session A/B on that 100-tool surface, driven by the corpus's own queries, took 13.4% off the whole bill.

And that run is the one that found a cost. xLAM ships an answer key, so it is the first run that could ask whether the saving bought worse answers, and it did: the routed arm called every tool the key names on 75.6% of sessions against the control arm's 87.2%, p = 0.003. It is not a mystery. On a wide surface the fixed menu pool cannot cover the traffic, and the share it misses at publish time is close to the share of requests the agent then fails. Pruning per session instead removes the cost entirely, at p = 0.71, and removes eleven of the thirteen points of saving with it, because a prefix no other session shares is a prefix that never caches. That trade is yours to make and both sides of it are measured in docs/scenarios.md.

It tunes itself. scripts/tune_settings.py sweeps the menu size, the menu count and the miss price against the agent's own logged history, prices each candidate on sessions it was not fitted to, and writes down what it chose with the evidence beside it. It arrives at the same menu pool the reference agent ships, and it refuses to recommend a change that does not replicate on both halves of the held-out data.

And it keeps itself current. Two daily jobs close the loop. One retrains when the model stops fitting the tools it is serving and publishes only if the new model beats the one already live. The other compares the live generation against the holdout arm and withdraws it if routing is costing more than not routing, which every running process picks up within about a quarter of an hour, with no redeploy. What that buys is a floor rather than a promise: the worst outcome is that nothing improves, and a saving that becomes available reaches the agent without anyone doing anything. Turning the router on is still yours to do, and so is deciding the holdout size.

It also routes on day zero, and survives newly deployed tools. Setting judges=("prototype",) or judges=("xgboost", "prototype") (SMART_ROUTER_JUDGES) activates PrototypeJudge, which embeds the live tool docstrings and JSON schemas once per process, subtracts the catalog mean vector, and propagates semantic activation one hop upstream across the Returns: $\to$ Args: prerequisite graph. With zero training rows and no GCS artifact, PrototypeJudge reaches 92.6% recall (91.0% complete sessions, 89.0% token efficiency) on the multi-hop support agent and 98.9% recall (97.8% complete sessions, 88.0% token efficiency) on the 2,000-session Salesforce xlam corpus at k=15 (or snaps to zero-data canonical menus synthesized from the tool catalog via cold_start_menu_count=24). Paired with the trained XGBoost booster (judges=("xgboost", "prototype")), both judges share the single turn-one prompt embedding (225.7 ms G3 total latency) while PrototypeJudge scores classless and newly deployed tools semantically instead of carrying every unseen tool on every menu (scripts/score_sets.py --compare-tiers):

When 15 active tools are held out from xgboost to simulate newly deployed tools (--unseen-holdout 15), pruning unseen tools under xgboost alone drops support recall to 0.6617 (62.9% complete sessions), whereas hybrid (xgb+proto) rescues 0.9004 recall (88.4% complete sessions, 0.9483 token efficiency), and on Salesforce xlam lifts pruned-session recall from 0.9512 (91.8% complete) to 0.9784 (96.1% complete; 0.9815 overall).

What is not proven. The saving is measured on one agent, and that agent was built by this project to have a realistic tool surface. The number most likely to move is the cache behaviour, which depends on how densely your sessions arrive. The known limit is the menu pool: the fixed set of menus that keeps the prompt cache warm stops covering the traffic somewhere past a hundred tools, and on the widest third-party surface it recalls 0.23 where the bespoke prediction recalls 0.97. If your catalogue is that large, the router still predicts well and you would run it without the pool, which is measured above and saves 2.0% rather than 13.4%. The honest way to find out which side you are on is the same A/B this repo runs, and the plugin ships with a permanent holdout arm so the comparison keeps running after rollout.

docs/evidence.md is this section on one page, with the script and the write-up behind every number in a table at the end.

Training needs a corpus, BigQuery, and a Gemini Enterprise embedding model, and generating a corpus costs real money. See docs/training.md for what that involves. The plugin ships with mode at "off" and route_fraction at 0.0, so installing it changes nothing until you turn it on, and a built-in guard refuses to prune on any agent whose prompts are too small for pruning to be worth it.

How it works

The router is one ADK plugin, installed on the Runner. On the first turn of a session it embeds the user's message, asks a small gradient-boosted classifier which tools are likely to be needed, and filters the declared tool list down to those. It then keeps that decision for the rest of the session and stops paying for it.

It does not collect its own training data. It reads the table ADK's BigQuery Agent Analytics plugin (ADK integration guide, BigQuery documentation) is already writing, so an agent that logs its turns has, without doing anything else, the corpus this router trains on. An agent that has been logging for a month can train on that month.

Every arrow out of the Runner is a separate call. The two plugins never call each other. They meet twice, on objects the Runner hands to both of them, and the notes mark those two meetings.

The diagram is one turn's model call and no more. It leaves out the router's other hooks: after_model_callback, where it folds the model's reported token counts back into its guards, and the before_tool and after_tool pair that runs the tool result cache described below. It also undersells the analytics plugin, which implements all fourteen ADK callbacks and writes rows from several of them. The router's own outward calls are the bounded part: one embedding call on the first message of a session, and one artifact fetch per process.

Training happens between deployments, on rows the analytics plugin has already written.

Plugin order is load-bearing rather than stylistic. An ADK plugin is an object with up to fourteen callbacks, and the Runner runs a given callback on every registered plugin in turn, in the order you registered them. These two meet at two of those callbacks, and the router has to go first at both. In before_model_callback it edits the tool list that the analytics plugin then reads out of llm_request.tools_dict, so the wrong order logs a menu the model never saw. In on_event_callback it stamps the routing decision onto event.custom_metadata, which is the field the analytics plugin copies into attributes.custom_metadata, so the wrong order logs nothing about the router at all. The second is the quieter failure: the agent works, the rows arrive, and every measurement in this README silently becomes unavailable.

Session stability is the whole design, not an optimisation. Gemini caches the prefix of a request, and the tool declarations sit inside that prefix. Re-deciding every turn would rewrite the prefix every turn, break the cache every turn, and cost more than the pruning saves. The router decides once, and if it is not confident it declares everything and gets out of the way.

That is also what decides which agents this suits. The classifier sees one message, the first one, so it fits an agent whose users open by saying what they want. "Where is my order" is routable. An agent whose sessions start with a greeting and reveal the intent three turns later is not, and neither is one where the useful message is "what about the other one?", because the router has already decided and locked by then. Those sessions are not broken, they are declined: an unconfident prediction declares every tool and the agent behaves exactly as it would without the plugin. So the cost of a bad fit is that nothing happens, which is the right cost for a guess made this early. On the reference corpus a little under a third of sessions are declined this way.

For the same reason the router does not assemble a fresh set of tools per session. Training picks a handful of fixed menus from the corpus and ships them in the model artifact; at decision time the classifier's probabilities only choose which menu to send. Sessions that share a menu share a prefix, and a shared prefix is one that caches.

The second half is a tool result cache. You declare which tools are safe to memoize and for how long, and repeat calls with the same arguments are answered from memory. The cache key always includes the caller's identity, because most real tools are scoped to the principal and take arguments that never mention them. get_balance() returns different answers for different people.

Installing

Requires Python 3.11+, uv, and Application Default Credentials (gcloud auth application-default login) for cloud commands.

uv add adk-smart-router
# or directly from GitHub:
uv add git+https://github.com/statmike/adk-smart-router
uv run smart-router qualify --app myapp.agents:root_agent

If you use an AI coding assistant (Gemini CLI, Claude Code, Antigravity, Cursor, or Codex), you can also install the bundled Agent Plugin skill so your assistant can qualify, wire, and configure the router in your project:

npx skills add statmike/adk-smart-router

Ask that qualify question before anything else. The smart-router command answers five, in the order you hit them:

qualify reads your agent's own declarations and prompt size, with no cloud account and no credentials involved. TRAP is the one verdict that means do not install: your prompt caches today and a prune would carry it under the model's cache floor, which measured +25.5% the one time it happened. Pass --history-tokens with what a real turn of yours carries, because that is what lets the command tell your two cache floors apart, the whole prompt within a session and the instruction plus declarations across them.

It also answers the question adopters usually hit a month later: how much of your tool surface a turn-one router can reach at all. A tool is predictable from the first message only if that message can supply what the tool requires, so anything needing an id your systems assign is a second hop that no prediction reaches. That part is not lost, it is carried by the menus instead. Nothing is guessed from argument names, because order_id and subscription_id look identical and are not, so the cold answer is a range. To turn it into a number, run --write-arguments args.txt, mark each argument held or supplied, and pass it back with --arguments args.txt. The list is ordered so the first ten decisions settle the most tools, and examples/reference_agent/arguments.txt is one filled in.

readiness is the one to run next if your agent already logs to BigQuery Agent Analytics, because the router trains on history and nothing in the training run needs the router to have been installed while that history was written. It reads the log you already have and says whether it could train today, over four questions: do first messages carry text, are there enough labelled sessions over enough tools, does a session's tool use fit inside a menu, and would a plain frequency table already do the job. It needs read access and nothing else, no bucket and no connection, and it writes nothing.

doctor runs once the plugin is wired in and checks the misconfigurations that fail silently (plugin order, analytics allowlist, context cache config, and active judge dependencies / gcs_bucket when mode="active"). check-arguments holds your argument markings up against which tool each of a few thousand logged sessions actually opened with. It reports rather than decides: a first call proves the tool is reachable at turn one and does not prove your user held the argument, because the model normalises what it was told and sometimes invents what it was not, and the two look identical in a log. docs/install.md walks through all four.

status is the one you run afterwards, and it is described under Knowing whether it is paying below.

From a clone:

uv sync
uv run pytest

Using it

Install the plugin on your App, ahead of ADK's BigQuery Agent Analytics plugin:

from google.adk.agents import LlmAgent
from google.adk.agents.context_cache_config import ContextCacheConfig
from google.adk.apps import App
from google.adk.plugins.bigquery_agent_analytics_plugin import BigQueryAgentAnalyticsPlugin
from adk_smart_router import SmartRouterPlugin, load_config

app = App(
    name="support",
    root_agent=LlmAgent(name="support", model="gemini-2.5-flash", tools=ALL_TOOLS),
    context_cache_config=ContextCacheConfig(min_tokens=2048, ttl_seconds=1800),
    plugins=[
        SmartRouterPlugin(load_config()),
        BigQueryAgentAnalyticsPlugin(
            project_id="my-gcp-project",
            dataset_id="smart_router",
            custom_metadata_allowlist=["smart_router"],
        ),
    ],
)

Order matters. If an analytics plugin runs before the router, it logs every cache hit as a real tool call taking zero milliseconds, the measured tool lifetimes inflate without bound, and the next model trains on the router's own output. The router logs a warning if it detects this, but it cannot fix it for you.

load_config() reads the environment. To turn the router on for a tenth of sessions:

export SMART_ROUTER_MODE=active
export SMART_ROUTER_ROUTE_FRACTION=0.1
export SMART_ROUTER_GCS_BUCKET=your-bucket

To enable the tool result cache, declare which tools may be memoized and for how many seconds:

SmartRouterPlugin(load_config(cacheable_tools={"get_return_policy": 3600, "get_faq": 3600}))

Nothing is cached unless you name it here. Read-only-ness cannot be inferred from an ADK tool, and memoizing create_ticket silently drops the second ticket.

The public API is four names: SmartRouterPlugin, RouterConfig, ToolJudge and load_config. Everything else in the package is an implementation detail and may change.

Knowing whether it is paying

The router holds a fraction of sessions out of routing and logs both arms, so the saving is measured against a live control rather than estimated. One command reads it back:

$ uv run smart-router status --agent support

live artifact      20260905T171738Z    trained 2026-09-05, 3 days ago
  from run         f8897a5a-0419-498b-a873-25ab978d4b53  ·  9,043 rows / 6,789 sessions over 35 days
  gates            miss reduction 0.81  ·  menu recall 0.9659
  settings         24 menus of 11   (no tuning recorded)

last 7 days        1,479 sessions routed  ·  1,547 holdout  ·  1,851 in neither arm
  cost             G1 -11.0%   over a 1,547-session control arm
  decisions        holdout 32.7%  ·  reused 25.7%  ·  lowconf 17.6%  ·  late 8.3%  ·  floor 5.1%  ·  fail-open 4.4%
  tool cache       nothing served from cache, over 12,552 tool calls

drift              unseen share 27.4%  against a 26.1% baseline   (+1.3)
                   confidence p50 0.79  against 0.81
  verdict          not due, checked 2026-09-06

next retrain       2026-09-19, on age (max_model_age_days 14)
last cycle         2026-09-04   train ok  ·  tune declined  ·  publish ok

  read 0.25 GB

Read only, no model traffic, and it prints what it scanned. Every rate carries its denominator, and every number it cannot stand behind is an absence rather than a zero: with no holdout traffic the cost line says there is no control arm and prints no percentage at all.

It works on day one, when almost none of those numbers exist yet, because that is when somebody most wants to know whether anything is happening.

Reading it is the only part that needs you. Two scripts on a daily timer keep the loop turning: run_cycle.sh retrains when the model stops fitting the tools it is serving, and watch_generations.py withdraws a generation that turns out to cost more than not routing, which every running process picks up on its next poll without a redeploy. Both write a row every time they run, including the times they decide to do nothing, so a quiet loop and a stopped one do not look alike.

docs/observability.md covers the rest: the retrain ledger every training step writes to, the drift check, and which signals are allowed to say a retrain is due. docs/operations.md has the scheduler recipe.

Configuration

Every knob lives in RouterConfig in src/adk_smart_router/config.py, which is data only and documents each field where it is defined. The ones you are most likely to touch:

These can be set in code or through SMART_ROUTER_* environment variables. The operational ones are exposed as environment variables specifically so you can flip the router off without a deploy.

What is in the repo

The package modules, in the order a request touches them:

Eight of those never run on a request: reachability.py, openings.py, qualify.py, readiness.py, generation_watch.py, lifecycle.py, status.py and cli.py. They are the offline half, what a retrain cycle writes down and what reads it back.

The other twenty are the serving path, and they are exactly the import closure of plugin.py. Two of them look like they belong on the offline side and do not: plugin.py imports the configuration types directly, and model_store.py compares drift signals before it accepts an artifact. That boundary is where an extraction of the plugin alone would cut.

That paragraph is a test rather than a claim. tests/test_serving_seam.py reads the import graph out of the source and fails if the closure moves, in either direction, so a hot-path module cannot quietly pick up a dependency on something that reads the warehouse.

The rules it will not break

These are the constraints the design is built around. They are enforced in code and covered by tests, and they are the first things to check if you change anything.

The router never fails a request. ADK turns a plugin exception into a RuntimeError that aborts the whole run, so every callback swallows its own exceptions and returns "no opinion". A broken model artifact degrades to an unrouted agent, never to an outage.

Uncertainty widens the menu, never narrows it. When there is nothing to prune towards, every tool is declared, and a session that has sent the full list once keeps sending it: its prefix is set, and swapping in a shorter one now would buy a smaller prompt at the price of a new one. Doubt on its own is not a veto, though: a shaky prediction that removes a lot of tokens is still worth sending, and what settles it is the arithmetic in the next paragraph rather than a confidence threshold.

A menu is chosen once per session and never widened. There is no mid-session correction, so a session whose menu lacks a tool it wanted stays without it. That is priced up front rather than repaired: it is exactly what miss_cost_tokens buys.

Pruning happens only when it pays. Before a menu goes out, the router compares what the removed declarations save over an expected session against the chance the session reaches for a tool the menu dropped, priced at miss_cost_tokens. Sessions that fail that comparison get the full list and are recorded as cheap.

The cache key always contains the caller's identity. There is no opt-out and no shared scope. A tool that writes to tool_context.actions is blocklisted the moment it is caught, because a cache hit reproduces the return value and none of the side effects.

An agent that gates its own tools is never cached. A plugin-level cache hit short-circuits before the agent's own before_tool_callback runs, and that is where authorization usually lives.

Nothing is cached across users, and nothing is speculatively executed. Running a tool the model has not asked for would be the fastest thing in here and it is deliberately absent: only a quarter of opening messages carry the arguments their first tool call requires (scripts/price_prefetch.py).

docs/threat-model.md is these rules from an attacker's side: what someone who writes the first user message can and cannot make the router do, and the three things an operator has to get right.

Pruning stops when it starts costing more than it saves. Gemini bills a cached input token at about a tenth of an uncached one, so a prune that pushes the prompt under the model's context-cache floor makes every token it kept ten times more expensive. Two guards prevent that: one measures the request before shrinking it, and one watches what actually cached and turns pruning off for an agent that is losing hits by it. This is not a hypothetical. See the cache guards.

Developing

uv sync
uv run pytest              # 1,900+ tests, all offline
uv run ruff format --check .
uv run ruff check .
uv run mypy
./scripts/check_install.sh # build the wheel, install it, run the first minute

The test suite makes no network calls and needs no credentials. Anything that touches GCP lives in scripts/ and is run deliberately.

Those scripts have no default project and no default Cloud Storage bucket. Point them at your own deployment once, in your shell profile, and every script and both example agents pick it up:

export SMART_ROUTER_BQ_PROJECT=your-gcp-project
export SMART_ROUTER_GCS_BUCKET=your-bucket

Leave them unset and each script says which variable is missing rather than failing somewhere inside a project you do not own. Nothing in this repository names a project.

check_install.sh is the one gate that does not point at the source tree. It builds the wheel, installs it into an empty environment, and runs the subcommands against examples/tiny_agent, which is the first minute of using this. It costs nothing and runs in CI on every push.

Further reading

Built from the AI-Native Project Harness.

Versioning

1.0.0, and the promise is narrow and worth stating. It covers the four names you import: SmartRouterPlugin, RouterConfig, ToolJudge and load_config. Those keep their shapes across 1.x, and a change that breaks them is a major version. The configuration keys and the smart-router subcommands follow the same rule; anything under a leading underscore does not.

What 1.0.0 does not promise is the saving. −22.5% was measured on one agent with authored traffic, and the honest reading of it is "this worked here". Your number comes from your own holdout arm, which ships enabled.

License

Apache 2.0. See LICENSE.

This is not an officially supported Google product. It is built on Google products and is neither endorsed nor supported by Google.