lap — token-efficiency scorer for agent-facing APIs

lap is the open, neutral, standalone toolkit (no pet-zoo dependency) that measures how many tokens an API's definitions cost an LLM. It answers: is my agent-API menu efficient, and by how much could it shrink? — an open, reproducible number to set beside the fast-growing MCP/OpenAPI tooling (see ../docs/LANDSCAPE.md for the neighbors LAP builds on and credits).

Install

pip install -e .                  # from the repo root (or: pip install lap-score once published)
pip install -e ".[mcp]"           # + real-MCP baseline (fastmcp)
pip install -e ".[faithful]"      # + faithful Anthropic count_tokens

Core deps are just httpx + tiktoken + pyyaml; fastmcp and anthropic are optional extras. Robust to real specs: allOf/oneOf/anyOf, $ref in params/requestBodies/responses, path-item-level parameters, OpenAPI 3.1 type lists, external $refs (left intact), YAML input, Swagger/OpenAPI 2.0 (response schema, in: body params, type-on-parameter, #/definitions), and non-JSON media types (*+json, form, XML). Verified crash-free + non-degenerate across 175+ real APIs.guru specs — re-run with ../experiments/fuzz_corpus.py.

Quickstart

lap score  https://petstore3.swagger.io/api/v3/openapi.json   # menu (bucket A) token cost
lap lint   https://petstore3.swagger.io/api/v3/openapi.json   # flag LAP rule violations
lap score  --mcp-url http://localhost:8080/mcp                # score a live MCP server's tools
lap lint   --mcp-url http://localhost:8080/mcp                # lint a live MCP server (M-rules + grade)
lap lint   --mcp "python -m mcp_server_git"                   # ...or over stdio
lap score  lap/examples/bookstore.openapi.json

# no install needed, from the repo root:
python -m lap.score lap/examples/bookstore.openapi.json

Example output:

LAP menu score - Bookstore API
operations: 6   referenced component schemas: 2

  variant       A tokens  saved vs full  form
  openapi_full  418       +0%            6 tool(s)
  compact_sig   205       +51%           manifest text
  numbered      168       +60%           manifest text

Menu efficiency: compact signatures are +51% vs naive OpenAPI->tools (418 -> 205 tokens).

When fastmcp is installed, the score also includes a real-MCP baseline (FastMCP.from_openapi) — what an actual MCP generator emits — plus its output-schema-inclusive figure (--no-mcp to skip). On a real public API (Swagger Petstore, 19 ops) the real MCP server costs 2226 menu tokens (3844 with output schemas) vs 415 for compact signatures — an ~81% reduction. The toy finding holds in the wild: a real MCP generator is heavier than the naive baseline.

The score also includes a lazy tool_search form (the Anthropic Tool Search / Cloudflare Code Mode pattern: a fixed 2-tool menu + a name index, schemas loaded on demand). Because it doesn't preload schemas, its bucket A is ~flat in the number of operations — on a 120-operation API it collapses the menu ~83% vs full schemas, beating even compact signatures at scale (Petstore: 1835 → 207, −89%).

• Faithful counts: set ANTHROPIC_API_KEY (uses the free Anthropic count_tokens endpoint; tool defs counted via the real tools= parameter). Without it, a GPT-style tiktoken approximation — absolute numbers approximate, relative ordering robust.

The LAP grade + badge

lap score also prints a composite LAP grade — one documented 0–100 number (letter A–F) folding three sub-scores: menu (naive-menu tokens per operation, 0.45), result (heaviest estimated response, 0.30) and hygiene (lint findings per operation, 0.25); log-scaled, constants in grade.py, formula in the profile. Calibration on real APIs (post-M3, query params counted): LaunchDarkly B, Spotify/SQS C, GitHub/Postman/DynamoDB D, Google Drive F.

lap badge turns it into a README badge (shields.io endpoint JSON):

lap badge api/openapi.json -o docs/lap-badge.json   # commit it, then embed:
# https://img.shields.io/endpoint?url=<raw URL of docs/lap-badge.json>

…or from the bundled Action, add badge-path: docs/lap-badge.json.

Score your installed MCP stack

lap stack answers the 2026 headline question — "how many tokens does my agent pay before I type a word?" — for your machine. It reads the agent's own MCP config (Claude Code project .mcp.json, Claude Code ~/.claude.json, Claude Desktop claude_desktop_config.json, or any JSON with an mcpServers map), connects to every server it lists (stdio or HTTP), and totals the advertised tool menus:

lap stack                        # auto-discover Claude Code / Claude Desktop configs
lap stack path/to/mcp.json       # or score an explicit config
lap stack --only github,jira     # subset; --json for machine-readable; --timeout N per server

LAP stack scan - tokenizer: tiktoken-approx

demo-mcp-config.json
  server        kind   tools  menu tokens  compact  note
  time          stdio      2          283       31
  git           stdio     12         1418      153
  needs-node    stdio      -            -        -  RuntimeError: Client failed to connect: ...
  TOTAL                    14         1701      184

Your agent pays ~1,701 tokens of tool menus at session start - before you type a word
(14 tools across 2 reachable server(s)).
Compact signatures of the same tools would cost 184 (+89% saved); one lazy tool_search menu
over the whole stack, 193 (+89% saved).

Unreachable servers (missing binary, no credentials) become annotated rows, not crashes. The stack-level tool_search what-if is counted honestly: the fixed search/call tools are paid once for the whole stack, plus a name index across all servers. Needs the [mcp] extra.

Auto-fix as an OpenAPI Overlay

lap fix turns the structurally fixable lint findings into an OpenAPI Overlay 1.0.0 document — advice becomes an applicable patch: R3 → a limit parameter, R1 → fields, R2 → filter, E1 → a declared 4XX error response. (D3/A1 stay advisory — renames and new endpoints are semantic decisions.) The overlay declares the contract; your server still has to implement it.

lap fix api/openapi.json -o lap-overlay.yaml          # apply with any Overlay-aware tool
lap fix api/openapi.json --apply patched.json         # ...or the built-in merge

On the bundled Bookstore example: 15 lint findings → 3 after applying, and the LAP grade jumps B (72) → A (91). (The menu forms currently count path+body parameters only, so the added query params don't change bucket A — see the roadmap for the query-param menu fix.)

Diff mode

lap score --diff <before> <after> compares two versions of a spec instead of scoring one — "did this PR make the API worse for agents?" Reports the menu token delta per form, plus which LAP lint findings were newly introduced or fixed:

lap score --diff old_openapi.json new_openapi.json
lap score --diff old_openapi.json new_openapi.json --gate-form compact_sig --max-growth 500

CI gate

--json makes both commands machine-readable; thresholds set the exit code so LAP can fail a build:

lap score openapi.json --gate-form compact_sig --max-menu-tokens 800   # exit 1 if the menu is too heavy
lap score --diff old.json new.json --gate-form compact_sig --max-growth 500  # exit 1 if the menu grew too much
lap lint  openapi.json --fail-on warn                                  # exit 1 on any warning
lap lint  openapi.json --ignore R2,A1                                  # suppress rules (or a ./.lapignore file)

GitHub Actions:

- run: pip install lap-score          # or: pip install -e .
- run: lap score api/openapi.json --gate-form compact_sig --max-menu-tokens 800
- run: lap lint  api/openapi.json --fail-on warn

…or the bundled composite Action (one step, no manual install):

- uses: lCrazyblindl/lap@v0.5.0
  with:
    spec: api/openapi.json
    max-menu-tokens: "800"     # gate the compact_sig menu (omit = report only)
    fail-on: warn              # fail on any lint warning (omit = report only)

Already lint OpenAPI with Spectral? The same LAP rules ship as a ruleset — see ../spectral/.

What it measures (and what it doesn't)

It measures bucket A (the definitions/menu the model carries in context) and estimates the other two from the schemas: B (the call the model emits — tool name + required args in a minimal tool-use envelope; optional params omitted, real schema examples honored) and C (result size, from each response schema + an assumed --page-size). Both estimates are structural lower bounds that capture keys/nesting/types. For list responses the C table also shows a projected figure — the same page with each item cut to its first 3 schema fields — annotated with whether the API actually advertises a projection param (fields=/$select), so rule R1's saving is a number per endpoint, not advice. For measured B/C on live tasks see ../experiments/token-bench. The conventions behind the compact form are the LAP profile.

Files