tabagent 0.1.0

Replace an LLM agent's closed-set decisions with a cheap classifier trained from its own logs.

TabAgent: Distilling LLM Agent Components into Cheap Classifiers
_{Ido Levy^1,2 · Segev Shlomov¹ · Avi Yaeli¹ · Nir Mashkif¹ · Asaf Adi¹ · Yinon Goldstein¹ · Eilam Shapira¹ · Leshem Choshen^1,2
1IBM · ²Weizmann Institute of Science}

_{▶ Watch the 73-second explainer}

Quick start · How it works · Results · Use it in your agent · Docs · Cite

Why

• Agents pay a frontier LLM for the same kind of decision again and again. Which tools to shortlist, which skill or application to use, which model or sub-agent to route to: closed-set decisions whose repeated calls dominate per-task latency and cost.
• A classifier trained on the agent's own successful runs can make them instead. On AppWorld tool shortlisting it recovers essentially all of the tools the GPT-5.4 shortlister used on its successful runs (Recall@9 of 92–100), in 2.682 ms for $2.0×10⁻⁷ per decision against 7.50 s and $0.052: more than 10³× lower latency and 10⁵× lower cost.
• The representation does the work, not the model. A head over raw text or embeddings alone collapses; trace-derived features make it accurate, so one lightweight CatBoost head serves every decision family.

How it works

	Discovers features. LLM agents analyze the successful trajectories, write and sandbox-test the extraction code, and a judge ensemble keeps the features that matter.
	Augments the rows. Deterministic, label-preserving paraphrases of the free-text state, where trajectories are scarce.
	Decides in milliseconds. A CatBoost classifier scores every candidate in one pass; a confidence gate can defer unsure decisions back to the LLM.

The three phases run once per agent and catalog. At inference the head replaces the LLM call, and the rest of the agent is unchanged. Components covers each phase in detail.

Quick start

pip install tabagent

See it work on a public benchmark, in a few minutes on a laptop CPU (it downloads 3.5 MB of data and a 128 MB encoder once):

git clone https://github.com/dolev31/TabAgent && cd TabAgent
pip install -e ".[embed]"
python examples/01_quickstart_skillsbench.py   # P@R ≈ 0.80, Recall@9 ≈ 0.90 on SkillsBench skill selection

The paper reports P@R 80 and Recall@9 91 there; Reproducing the paper compares every public result with the paper's.

pip install "tabagent[tabschema]"   # onboarding from agent logs (OpenAI and OpenAI-compatible LLMs)
pip install "tabagent[llm]"         # Anthropic, Gemini and other providers through LiteLLM
pip install "tabagent[embed]"       # e5 context-candidate similarity features
pip install "tabagent[serve]"       # the HTTP decision service

Results

_{AppWorld tool shortlisting: TabAgent (star) is the fastest, cheapest point, near the top on quality.}

More findings

• In the loop. Hot-swapped into a live agent on 24 held-out AppWorld tasks, TabAgent reaches 27.1% task success against 29.2% for the GPT-5.4 shortlister; the paired 95% CI of the difference, [−18.8, +16.7], includes zero.
• Few traces. The head reaches 90% of its full-pool P@R from 118 successful trajectories on AppWorld and from 37 to 51 on the four τ²-bench agents.
• Across agents. Trained on one ReAct agent's traces and tested on another's, it loses about 3.1 points of P@R on τ²-bench (transfer within one architecture).
• The head barely matters. Every classifier head reaches useful P@R with the feature card and collapses to 0.10–0.21 without it; as the candidate set grows, LLM regimes degrade sharply while TabAgent stays highest.

Reproduce the paper

Every public result has one command; make reproduce runs every public Table 1 and Table 2 cell on a CPU.

Reproducing the paper lists every table and figure with its command, time and data, puts the released numbers next to the paper's, and names what public data cannot reproduce (the results on the CUGA agent's AppWorld trajectories).

Use it in your agent

1. Onboard a decision from the agent's own logs (OpenAI, Anthropic, LangChain, τ²-bench, CUGA or generic JSON):

export OPENAI_API_KEY=...                                          # the LLM TabSchema calls
tabagent scan ./agent_logs                                         # which LLM calls are worth replacing?
tabagent onboard ./agent_logs --decision ToolShortlist --dry-run   # what it will process and cost
tabagent onboard ./agent_logs --decision ToolShortlist --out ./my_head

2. Serve it: tabagent serve ./my_head --port 8077 answers POST /decide from warm, sandboxed workers, with readiness, Prometheus metrics and drift monitoring (production runbook).

3. Hot-swap the LLM call: ToolFilter sends your tool-calling LLM only the tools the head shortlists, and the full list whenever the head defers.

from tabagent.integrations.openai_tools import ToolFilter

tool_filter = ToolFilter("my_head", k=7)  # or the URL of a `tabagent serve` service
session = tool_filter.session(task={"instruction": goal})  # one per conversation
client.chat.completions.create(model=..., messages=messages, tools=session(messages, tools))

import tabagent

head = tabagent.onboard("./agent_logs", decision="ToolShortlist")
print(head.report)  # grouped cross-validation vs BM25 and chance, latency, cost per decision
head.save("my_head")
head.decide(prefix, candidates, k=5)  # {"candidates", "scores", "confidence", "deferred", "error"}

from tabagent import TabAgentRanker

# one row per (decision, candidate); y = 1 if the agent chose the candidate
ranker = TabAgentRanker(group_col="task_id", candidate_col="tool")
ranker.fit(X, y)  # infers the card, picks TabSynth and the prior by nested validation
ranker.rank(X_new, k=5)  # top-5 candidates per decision

Documentation

Concepts · Components · Onboard an agent · Bring your own table · Deploy · Datasets · Reproducing the paper

The guides are also a searchable site at dolev31.github.io/TabAgent. Pretrained heads for SkillsBench skill selection and τ²-bench tool shortlisting load with tabagent.load("hf://..."), and the Colab quickstart runs one in the browser.

Citation

@misc{levy2026tabagent,
  title  = {TabAgent: Distilling LLM Agent Components into Cheap Classifiers},
  author = {Levy, Ido and Shlomov, Segev and Yaeli, Avi and Mashkif, Nir and Adi, Asaf and
            Goldstein, Yinon and Shapira, Eilam and Choshen, Leshem},
  year   = {2026},
  eprint = {2602.16429},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  doi    = {10.48550/arXiv.2602.16429},
  url    = {https://arxiv.org/abs/2602.16429}
}

License

Apache-2.0 (see LICENSE). TabAgent ships no benchmark data and no agent trajectories; the benchmarks keep their own licenses, listed in NOTICE. Contributions are welcome: see CONTRIBUTING.md and the Code of Conduct; SECURITY.md covers how generated extraction code is sandboxed and how to report a vulnerability privately.

Acknowledgements

The public results are recomputed from the benchmarks' own releases, downloaded at pinned revisions and verified by SHA-256: τ²-bench and the tau2-sft-final trajectories, SkillsBench, Mind2Web (CC-BY-4.0), LLMRouterBench and AppWorld. The paper also evaluates on AppWorld with the CUGA agent and on the OAK and BPO benchmarks. TabHead is built on CatBoost, the similarity features on e5-small-v2, and the brand type is Inter and JetBrains Mono (SIL OFL 1.1).