API Agent

Load an API description — OpenAPI/Swagger or GraphQL natively, a Postman collection / RAML / API Blueprint (auto-converted on load), or even a prose API reference doc — ask a question in natural language, and the agent picks the right operations, calls them (strictly read-only), and returns an answer with citations plus a multi-signal evaluation (grounding, sufficiency, responsiveness). Ships with a Streamlit UI with live steps, streaming answers, and token/cost tracking.

Load spec(s) / reference doc ──▶ Catalog of operations
                     │
User question ──▶ Router + intent (narrow to relevant ops) ──▶ Executor (call operations)
                     ──▶ Synthesis + Citation ──▶ Self-review ──▶ Evaluator ──▶ Answer

The LLM provider is any Chat Completions-compatible endpoint, so the model is a config value — hosted or local providers are swappable without code changes. The generator, judge and router roles are configured separately. Full internals are documented in the source repository.

Install as a pip package (share it / minimal setup)

python -m venv .venv && source .venv/bin/activate
pip install "codi-api-agent[all]"                          # UI + embeddings
# pip install "codi-api-agent[all,sql]"                    # …and the Postgres backend
# or from a wheel someone shared with you:
# pip install "codi_api_agent-0.3.1-py3-none-any.whl[all]"
export LLM_API_KEY="your-provider-key"
api-agent                                                  # opens the UI at http://localhost:8501

The distribution is codi-api-agent; the import name stays api_agent (from api_agent import Agent).

Then add a free no-auth demo spec in the sidebar (Countries GraphQL or the SWAPI Postman collection) and ask away. Full step-by-step in TUTORIAL.md. Build the wheel yourself from a checkout with pip install build && python -m build (→ dist/).

Setup (from a source checkout)

python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"       # editable install with UI + embeddings
# create .env with your provider + key, e.g.
printf 'LLM_BASE_URL=https://your-provider.example/v1\nLLM_API_KEY=your-key\n' > .env

Point LLM_BASE_URL at any Chat Completions-compatible endpoint and set LLM_API_KEY. A locally hosted endpoint works the same way — use any non-empty string as the key if it does not check one.

Set GENERATOR_MODEL / JUDGE_MODEL to models your provider serves that support tool/function calling.

Rate-limit resilience: give several keys (LLM_API_KEYS=key1,key2,…) and/or whole backends (LLM_POOL='[{"base_url":…,"api_key":…,"model":…}, …]') and the client rotates over them per query and fails over on a 429 with a cooldown — useful on free tiers.

Cost controls (for paid, per-token providers): set MAX_RESPONSE_TOKENS to cap tokens per response — the agent stops early and returns a partial answer once the ceiling is hit (0 = unlimited). Each response also shows an estimated $ cost; override the built-in per-model prices with MODEL_PRICING (JSON per 1M tokens, e.g. MODEL_PRICING='{"your-model":[2.5,10]}') or the sidebar 💲 Budget & cost fields. Unknown/free models simply show no cost.

Run

api-agent            # console script installed with the package

It opens empty — add an API via the sidebar's 📚 Load an API panel (set DEFAULT_SPEC=<url-or-path> in .env to auto-load one on startup). Free, no-auth demos to try: the Countries GraphQL endpoint https://countries.trevorblades.com/ (tick GraphQL API), or the Petstore spec https://petstore.swagger.io/v2/swagger.json → “fetch all pets that are sold”.

Load any API spec

Sidebar → 📚 Load an API → paste a spec URL or file path (or upload files) → ➕ Add spec. OpenAPI/Swagger and GraphQL load natively; a Postman collection, RAML 0.8, or API Blueprint file is detected from its content and auto-converted to OpenAPI on load. You can add several sources (any mix of formats) — each keeps its own base URL and auth, and one question can span all of them.

Every GET/HEAD operation (or GraphQL query field) becomes a callable tool (write operations are excluded — the read-only guardrail; they remain describable in documentation mode). Each operation maps: operationId → name, summary/description → routing text, parameters → arguments, servers (or Swagger-2.0 host+basePath) → base URL.

Authenticated (private) APIs: open the Auth (optional) expander and set a header before loading — e.g. Authorization = Bearer <token>, or X-API-Key = <key>. It's attached to every call. If an API needs auth and none is set, the agent says so instead of guessing.

No spec? Write a small one by hand — describe just the GET endpoints you care about (copy a block per endpoint), then load it as a file path. You don't need to be an OpenAPI expert; the summary/description you write are what the agent routes on.

Other formats (RAML / API Blueprint / Postman)

The agent's pipeline is format-agnostic — only the loader speaks OpenAPI — so these are converted to OpenAPI automatically when you load them (in the UI or via load_catalog). You can also convert ahead of time with the upstream npm tools directly (apib2swagger, api-spec-converter, postman-to-openapi).

Requires Node/npx (the converters are npm tools, fetched on first use).

No spec at all? Load a prose API reference doc

Sidebar → 📚 Load an API → source type API reference doc → point it at a reference page (URL, file, or upload) → 🔍 Extract endpoints. The documented METHOD /path lines, curl examples, path and query params are extracted from the doc's literal text (free, deterministic — it cannot invent an endpoint); an optional checkbox lets the LLM also enrich param types or handle prose-only docs. You then review and approve the extracted endpoints before any become callable — GETs load as tools, writes as documentation-only.

Inspect what a spec produces from Python:

from api_agent.graphql_loader import load_catalog
catalog = load_catalog("your_spec.yaml")
print(len(catalog.tools), "callable operations")

Connect a Postgres warehouse (SQL backend)

A data source does not have to be an HTTP API. Point the agent at a Postgres schema and every function in it becomes a read-only tool — the same pipeline (route → execute → synthesize → evaluate), minus the network hop.

pip install "codi-api-agent[sql]"          # adds psycopg; NOT included in [all]
export SQL_DSN="postgresql://readonly_user:pass@localhost:5432/warehouse"

Then Sidebar → 📚 Load an API → Postgres warehouse, or set SQL_DSN and it loads on start. Parameter names and types come from the database catalog itself, so there is no type inference to get wrong, and descriptions come from each function's COMMENT ON FUNCTION.

Read-only by construction, three independent walls:

1. only introspected functions are callable — there is no tool that accepts a SQL string, so free-form SQL cannot be expressed;
2. every connection opens default_transaction_read_only=on with a statement_timeout;
3. point SQL_DSN at a role holding only SELECT/EXECUTE.

Arguments are bound server-side with an explicit ::type cast taken from the catalog — never interpolated into SQL text — and every call is LIMIT-capped.

Both backends can be loaded at once — a supervisor routes each question to exactly one of them.

Verified working spec URLs (no auth)

Query your own GitHub repos

GitHub publishes its OpenAPI spec, so you can ask about your account:

1. Load the public spec (sidebar → 📚 Load an API → URL or file path): https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.json It has ~624 GET operations; the default Max operations to load (1000) loads them all and the router narrows per query.
2. Set a token in the Auth expander before loading: header Authorization, value Bearer <your GitHub PAT> (a token with repo / read scope).
3. Ask: “show my repositories” → routes to GET /user/repos and lists your repos.

More once loaded + authed: “who am I on GitHub?” (GET /user), “list my open issues”.

The big GitHub spec takes a few seconds to fetch/parse on load. Without a token, authed endpoints return 401 and the agent tells you a key is needed.

Routing (large specs)

When a spec has many operations, the router narrows them to the most relevant per query before the agent runs: hybrid recall — lexical (idf-weighted, stemmed) fused with local embeddings (a small local sentence-embedding model, free/offline; catches paraphrases with zero shared words) via Reciprocal Rank Fusion — then a fast LLM makes the final pick and classifies the request intent (write/doc/data) in the same call. Falls back to lexical-only if embeddings are unavailable. Specs with ≤ router_min_tools (default 6) operations skip routing. Configure in the sidebar or via ROUTER_ENABLED / ROUTER_TOP_K / ROUTER_MODEL / ROUTER_MIN_TOOLS / EMBEDDING_MODEL.

Testing & performance report

The source repository carries a deterministic regression suite (no LLM, no network) and an eval harness that reports guardrail reliability, per-category accuracy, faithfulness, read-only/PII safety, abstention, latency and run-to-run consistency. Neither is part of the installed package.

Project layout

api_agent/
  config.py          # env-driven settings (provider, per-component models, router, budget)
  llm.py             # Chat Completions client + multi-LLM rotation + tool-call recovery
  openapi_loader.py  # OpenAPI/Swagger spec -> callable Tools; compaction + PII redaction
  graphql_loader.py  # GraphQL introspection/SDL + the load_catalog format dispatcher
  spec_convert.py    # Postman / RAML / API Blueprint -> OpenAPI (auto, via npx)
  doc_extract.py     # prose API reference doc -> draft OpenAPI (structural; LLM optional)
  catalog.py         # Tool + Catalog (operation registry + result cache + name recovery)
  router.py          # hybrid lexical+embedding routing per query
  schemas.py         # Evidence, Citation, ToolCall, Faithfulness, AgentResult, Usage
  agent.py           # pipeline: cache -> route -> execute -> synthesize -> review -> evaluate
  ui.py              # Streamlit app: live steps, streaming, Stop, token/cost, doc review gate

Configuration reference

Everything is env-driven (.env or the environment). Only LLM_BASE_URL + LLM_API_KEY are required; the rest have working defaults.

Warehouse settings are in the SQL backend section above.

Notes & limitations

• Read-only: only GET/HEAD operations are exposed; write endpoints are never called.
• Honest failures: if an operation needs a key (401/403), is unreachable, or the server returns 5xx, the agent reports that clearly instead of guessing.
• Not production-hardened: the loader fetches the given spec URL and calls endpoints as-is — SSRF egress controls and per-user credential scoping are follow-ups; fine for dev against trusted specs.
• Public demo servers (e.g. Petstore v3) are often flaky — prefer v2 / a spec whose server you control.