API Agent

Load an API description — OpenAPI/Swagger or GraphQL natively, a Postman collection / RAML / API Blueprint (auto-converted on load), or even a prose API reference doc — ask a question in natural language, and the agent picks the right operations, calls them (strictly read-only), and returns an answer with citations plus a multi-signal evaluation (grounding, sufficiency, responsiveness). Ships with a Streamlit UI with live steps, streaming answers, and token/cost tracking.

Load spec(s) / reference doc ──▶ Catalog of operations
                     │
User question ──▶ Router + intent (narrow to relevant ops) ──▶ Executor (call operations)
                     ──▶ Synthesis + Citation ──▶ Self-review ──▶ Evaluator ──▶ Answer

The LLM provider is an OpenAI-compatible endpoint, so the model is a config value — Groq, HuggingFace, Ollama and OpenAI are all swappable without code changes. The generator, judge and router roles are configured separately. Full internals in How_It_Works.md.

Install as a pip package (share it / minimal setup)

python -m venv .venv && source .venv/bin/activate
pip install "codi-api-agent[all]"                          # from a package index
# or from a wheel someone shared with you:
# pip install "codi_api_agent-0.3.1-py3-none-any.whl[all]"
export LLM_API_KEY="your-groq-or-openai-key"
api-agent                                                  # opens the UI at http://localhost:8501

The distribution is codi-api-agent; the import name stays api_agent (from api_agent import Agent).

Then add a free no-auth demo spec in the sidebar (Countries GraphQL or the SWAPI Postman collection) and ask away. Full step-by-step in TUTORIAL.md. Build the wheel yourself from a checkout with pip install build && python -m build (→ dist/).

Setup (from a source checkout)

python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[all]"       # editable install with UI + embeddings
cp .env.example .env          # then edit .env with your provider + key

Set GENERATOR_MODEL / JUDGE_MODEL to models your provider serves that support tool/function calling (e.g. on Groq, llama-3.3-70b-versatile).

Rate-limit resilience: give several keys (LLM_API_KEYS=key1,key2,…) and/or whole backends (LLM_POOL='[{"base_url":…,"api_key":…,"model":…}, …]') and the client rotates over them per query and fails over on a 429 with a cooldown — useful on free tiers.

Cost controls (for paid, per-token providers): set MAX_RESPONSE_TOKENS to cap tokens per response — the agent stops early and returns a partial answer once the ceiling is hit (0 = unlimited). Each response also shows an estimated $ cost; override the built-in per-model prices with MODEL_PRICING (JSON per 1M tokens, e.g. MODEL_PRICING='{"gpt-4o":[2.5,10]}') or the sidebar 💲 Budget & cost fields. Unknown/free models simply show no cost.

Run

streamlit run app/streamlit_app.py

It opens empty — add an API via the sidebar's 📚 Load an API panel (set DEFAULT_SPEC=<url-or-path> in .env to auto-load one on startup). Free, no-auth demos to try: the Countries GraphQL endpoint https://countries.trevorblades.com/ (tick GraphQL API), or the Petstore spec https://petstore.swagger.io/v2/swagger.json → “fetch all pets that are sold”.

Load any API spec

Sidebar → 📚 Load an API → paste a spec URL or file path (or upload files) → ➕ Add spec. OpenAPI/Swagger and GraphQL load natively; a Postman collection, RAML 0.8, or API Blueprint file is detected from its content and auto-converted to OpenAPI on load. You can add several sources (any mix of formats) — each keeps its own base URL and auth, and one question can span all of them.

Every GET/HEAD operation (or GraphQL query field) becomes a callable tool (write operations are excluded — the read-only guardrail; they remain describable in documentation mode). Each operation maps: operationId → name, summary/description → routing text, parameters → arguments, servers (or Swagger-2.0 host+basePath) → base URL.

Authenticated (private) APIs: open the Auth (optional) expander and set a header before loading — e.g. Authorization = Bearer <token>, or X-API-Key = <key>. It's attached to every call. If an API needs auth and none is set, the agent says so instead of guessing.

No spec? Write a small one from the minimal template — describe just the GET endpoints you care about (copy a block per endpoint), then load it as a file path. You don't need to be an OpenAPI expert; the summary/description you write are what the agent routes on.

Other formats (RAML / API Blueprint / Postman)

The agent's pipeline is format-agnostic — only the loader speaks OpenAPI — so these are converted to OpenAPI automatically when you load them (in the UI or via load_catalog). To pre-convert from the terminal instead:

python scripts/convert_spec.py path/to/api.apib  -o specs/api.openapi.json   # API Blueprint
python scripts/convert_spec.py path/to/api.raml  --base-url https://your-host.com   # RAML 0.8
python scripts/convert_postman.py your.postman_collection.json --base-url https://your-host.com

Requires Node/npx (the converters are npm tools, fetched on first use).

No spec at all? Load a prose API reference doc

Sidebar → 📚 Load an API → source type API reference doc → point it at a reference page (URL, file, or upload) → 🔍 Extract endpoints. The documented METHOD /path lines, curl examples, path and query params are extracted from the doc's literal text (free, deterministic — it cannot invent an endpoint); an optional checkbox lets the LLM also enrich param types or handle prose-only docs. You then review and approve the extracted endpoints before any become callable — GETs load as tools, writes as documentation-only.

Inspect what a spec produces from the terminal:

python scripts/load_openapi_demo.py                          # default: Petstore
python scripts/load_openapi_demo.py --source your_spec.yaml --no-call

Verified working spec URLs (no auth)

Query your own GitHub repos

GitHub publishes its OpenAPI spec, so you can ask about your account:

1. Load the public spec (sidebar → 📚 Load an API → URL or file path): https://raw.githubusercontent.com/github/rest-api-description/main/descriptions/api.github.com/api.github.com.json It has ~624 GET operations; the default Max operations to load (1000) loads them all and the router narrows per query.
2. Set a token in the Auth expander before loading: header Authorization, value Bearer <your GitHub PAT> (a token with repo / read scope).
3. Ask: “show my repositories” → routes to GET /user/repos and lists your repos.

More once loaded + authed: “who am I on GitHub?” (GET /user), “list my open issues”.

The big GitHub spec takes a few seconds to fetch/parse on load. Without a token, authed endpoints return 401 and the agent tells you a key is needed.

Routing (large specs)

When a spec has many operations, the router narrows them to the most relevant per query before the agent runs: hybrid recall — lexical (idf-weighted, stemmed) fused with local embeddings (all-MiniLM-L6-v2, free/offline; catches paraphrases with zero shared words) via Reciprocal Rank Fusion — then a fast LLM makes the final pick and classifies the request intent (write/doc/data) in the same call. Falls back to lexical-only if embeddings are unavailable. Specs with ≤ router_min_tools (default 6) operations skip routing. Configure in the sidebar or via ROUTER_ENABLED / ROUTER_TOP_K / ROUTER_MODEL / ROUTER_MIN_TOOLS / EMBEDDING_MODEL.

Testing & performance report

Two ways to check the agent is behaving:

./scripts/run_checks.sh              # deterministic regression suite (no LLM/network) — see tests/README.md
python evals/run_eval.py --offline   # reliability report (from the suite), zero cost
python evals/run_eval.py             # full report: reliability + live accuracy on public APIs (needs LLM_API_KEY)

The eval harness writes a stakeholder-facing evals/report.md + report.json covering guardrail reliability, per-category accuracy, faithfulness, read-only/PII safety, abstention, latency, and (with --repeats N) run-to-run consistency. Details in evals/README.md.

Project layout

api_agent/
  config.py          # env-driven settings (provider, per-component models, router, budget)
  llm.py             # OpenAI-compatible client + multi-LLM rotation + tool-call recovery
  openapi_loader.py  # OpenAPI/Swagger spec -> callable Tools; compaction + PII redaction
  graphql_loader.py  # GraphQL introspection/SDL + the load_catalog format dispatcher
  spec_convert.py    # Postman / RAML / API Blueprint -> OpenAPI (auto, via npx)
  doc_extract.py     # prose API reference doc -> draft OpenAPI (structural; LLM optional)
  catalog.py         # Tool + Catalog (operation registry + result cache + name recovery)
  router.py          # hybrid lexical+embedding routing per query
  schemas.py         # Evidence, Citation, ToolCall, Faithfulness, AgentResult, Usage
  agent.py           # pipeline: cache -> route -> execute -> synthesize -> review -> evaluate
  ui.py              # Streamlit app: live steps, streaming, Stop, token/cost, doc review gate
app/streamlit_app.py # entry point for the UI
scripts/load_openapi_demo.py  # CLI: inspect a spec + one live call
tests/               # deterministic regression suite (no LLM/network)
evals/               # reliability + live-accuracy report harness

Notes & limitations

• Read-only: only GET/HEAD operations are exposed; write endpoints are never called.
• Honest failures: if an operation needs a key (401/403), is unreachable, or the server returns 5xx, the agent reports that clearly instead of guessing.
• Not production-hardened: the loader fetches the given spec URL and calls endpoints as-is — SSRF egress controls and per-user credential scoping are follow-ups; fine for dev against trusted specs.
• Public demo servers (e.g. Petstore v3) are often flaky — prefer v2 / a spec whose server you control.