Inferrail

Decide, execute, and record AP invoice-exception recovery.

For one eligible invoice-extraction exception, Inferrail decides whether it gets one permitted machine retry or your established human-review path, executes the retry through a supported integration, and records the resulting cost and outcome — with invoice content and provider credentials staying in your own process the whole time.

AP invoice-exception recovery: 30-second demo

pip install inferrail
inferrail ap demo

Fixture-based, zero-key, no network call. Runs the real decision engine through all five core scenarios — an eligible exception recovered by one retry, an unsuccessful retry that falls back to human review, a case the policy routes straight to human review, a repeated request handled without re-executing anything, and inspecting the resulting decision/outcome records — then points you at inferrail ap report to inspect them yourself.

Want a real (billed) OpenAI call instead of fixtures, or to see how you'd wire in your own extraction pipeline and review queue? See examples/ap_invoice_exception_recovery/. Full contract — supported failure types, retry method, validation contract, human-review handoff, versioned policy config, persistence/idempotency, the hosted HTTP API — in docs/capabilities/ap-invoice-exception-recovery.md.

No claim of proven savings or customer adoption is made anywhere in this README — see that capability doc's "Pricing and performance assumptions," which labels every dollar figure as an explicit assumption, not a validated result.

Also in this package: the self-hosted LLM gateway and cost receipts

Everything below this point is Inferrail's original, still fully-supported product: a self-hosted gateway that turns supported OpenAI chat-completion traffic into local, attributable economic receipts — the same receipt/cost substrate the AP product's OpenAIRetryAdapter and reporting build on. It remains available and unchanged; nothing about the AP release modifies its behavior or its commands.

Inferrail turns supported OpenAI chat-completion traffic into local, attributable economic receipts. Give related requests a customer-defined work_id, declare an outcome when your application knows one, and inspect the known inference economics associated with that work without storing prompts, responses, or tool payloads in Inferrail's own records.

For the supported chat-completions surface, Inferrail records known cost when measured usage and a verified price are available. Otherwise it reports unknown, never a fabricated $0.

Gateway: 30-second demo

pip install inferrail
inferrail demo

The demo needs no API key, no network call, and no provider billing. It runs canned requests through Inferrail's real engine with made-up prices labeled DEMO, then shows receipts, attribution, work-level economics, and explicit unknown evidence.

As of 0.2.0, the stable PyPI release includes the gateway, receipts, reports, TaskTransaction, and work commands (Work Economics), alongside AP invoice-exception recovery above.

What just happened?

AI request
  -> InferenceReceipt
  -> caller-supplied attribution
  -> related requests share work_id
  -> customer-declared outcome
  -> Work Economics

• Receipt: one inference request produced payload-free economic evidence.
• Attribution: the caller can attach identifiers such as customer, workflow, or project.
• Work: several requests can share a work_id that your application defines.
• Outcome: your application can append a declaration of what happened to that work.
• Work Economics: Inferrail joins that declaration with matching receipts and reports known attributed inference economics for the work.

You decide what a unit of work means: a contract review, support resolution, coding task, research run, or document-processing job. Inferrail associates economic evidence with the identifier your application supplies; it does not interpret the business meaning of that identifier or its outcome.

Request economics vs. Work Economics

Request economics: what known inference economics belong to one request?

Work Economics: what known inference economics belonged to the customer-defined unit of work those requests were performing?

This is not a full cost of work, COGS, margin, or business-value calculation.

Track a unit of work

The following uses real provider requests and requires OPENAI_API_KEY:

export OPENAI_API_KEY=<your-openai-api-key>

inferrail try "Review this contract clause" \
  -a work_id=contract_review_42

inferrail try "Identify remaining risks" \
  -a work_id=contract_review_42

inferrail work outcome contract_review_42 --status completed
inferrail work contract_review_42
inferrail work --all

For a gateway client, the equivalent generic attribution header is:

X-Inferrail-Attribute-Work-Id: contract_review_42

The deterministic offline demo includes this synthetic example:

work-contract-1
  2 inference receipts
  customer-declared outcome: resolved
  known attributed inference cost: $0.000483

resolved is only the demo application's own outcome meaning. Inferrail does not treat any outcome status as universally successful.

If Inferrail cannot verify the price for an observed inference event, its cost remains unknown rather than being treated as zero. No receipt evidence is also not the same thing as known zero cost.

First real request and reports

inferrail try is the shortest route to one real receipt. It uses your existing OPENAI_API_KEY; if it is not set, Inferrail prints what is required. It prints the response, receipt, measured tokens, known cost or unknown, the local receipt path, and the next report command.

inferrail try "Reply with one word: ready" --customer acme
inferrail report
inferrail report --by customer
inferrail report --by workflow
inferrail report --by provider

What a receipt contains

One payload-free JSON receipt per supported request:

{
  "receipt_id": "ir_1e6c916bac8940ca8a85",
  "provider": "openai",
  "model": "gpt-4o-mini",
  "prompt_tokens": 842,
  "completion_tokens": 191,
  "estimated_cost_usd": "0.000241",
  "attributes": { "customer": "acme", "workflow": "contract-review" }
}

(Trimmed — the full record also carries pricing provenance, status, route, timestamp, latency, and retry count. See Privacy boundary below for the complete shape.)

TaskTransaction: receipt-only task grouping

One task is rarely one call. Tag every request belonging to one unit of work with the same attribution value, then ask Inferrail what the task cost:

export OPENAI_API_KEY=<your-openai-api-key>
inferrail try "Reply with one word: ready" -a task_id=bug_9281
inferrail try "Summarize: the retry patch is deployed" -a task_id=bug_9281
inferrail transaction bug_9281

Task:        bug_9281
Transaction: tx_72fcfcca9ede9d2facc3
Status:      success

EVENT TYPE  EVENT ID                 STATUS   COST
inference   ir_f6fb6403d5324ea0acf9  success  $0.000003
inference   ir_756cc072a27f42f4a2ea  success  $0.000007

Known total cost: $0.00001

This TaskTransaction example uses real provider requests and a task_id. The offline demo instead correlates requests with work_id and shows Work Economics. Over HTTP, an X-Inferrail-Attribute-Task-Id: bug_9281 header does the same thing; inferrail.track_task(task_id=...) (see Attribute spend below) attaches it automatically to every nested call in an agent run, no header-threading required. See docs/adr/0008.

Use it as a gateway

For a long-running application, start the separate gateway process. The gateway process must have access to the provider credential through the configured environment variable; a key held only inside application memory is not automatically transferred to the gateway.

inferrail serve --quickstart

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Inferrail-Attribute-Customer: acme" \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'

The response is standard OpenAI choices/usage plus a non-standard inferrail block (route, provider, latency, retries) any OpenAI client already ignores. X-Inferrail-Attribute-* headers are optional attribution — never forwarded upstream. See examples/basic_chat_request.py for a minimal Python client, or point a supported OpenAI-compatible chat client at http://127.0.0.1:8000/v1. An OpenAI SDK client that does not set base_url can use its existing OPENAI_BASE_URL environment mechanism instead.

The default receipt is one JSONL line per supported request in ./inferrail-receipts.jsonl, relative to the gateway's working directory. Treat that file as machine/audit evidence; use inferrail report for the human aggregate, inferrail transaction <task-id> for receipt-only task grouping, and inferrail work <work-id> for work-attributed inference economics plus a customer-declared outcome.

# LangChain
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="not-needed",  # or your INFERRAIL_GATEWAY_TOKEN if auth is enabled
    model="default",
)

# LlamaIndex
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="default",
    api_base="http://127.0.0.1:8000/v1",
    api_key="not-needed",
    is_chat_model=True,
    context_window=8192,
)

# CrewAI
from crewai import LLM

llm = LLM(
    model="openai/default",  # "openai/" prefix required by CrewAI
    base_url="http://127.0.0.1:8000/v1",
    api_key="not-needed",
)

"model" normally selects a named route from inferrail.yaml (e.g. "default"), which maps to a provider + underlying model. If default_provider is set in your config, a model that matches no route is instead forwarded to that provider unchanged — so "model": "gpt-5.6-sol" works with no route pre-registered for it. Named routes always take priority. This passthrough is on by default for the zero-config quickstart path, off by default otherwise. Full design: docs/adr/0007.

Attribute spend

Three ways to attach business context to a request, all landing in the same attributes: dict[str, str] on its receipt:

• HTTP header (gateway): X-Inferrail-Attribute-<Name>: <value>, e.g. X-Inferrail-Attribute-Task-Id: bug_9281.
• CLI flag (inferrail try): --customer/--workflow shorthand, or generic -a <name>=<value> for anything else, including task_id.
• Ambient, for nested agent calls: inferrail.track_task attaches X-Inferrail-Attribute-Task-Id to every outgoing request for the duration of a with block or decorated function — no threading a task_id parameter through nested function signatures by hand.

import inferrail
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="not-needed",
    # also accepted by LangChain's ChatOpenAI, CrewAI's LLM, etc. via
    # their own http_client= argument
    # base_url must match the client's own base_url above — the header is
    # only ever attached to requests going to that destination.
    http_client=inferrail.attributed_http_client(base_url="http://127.0.0.1:8000/v1"),
)

@inferrail.track_task(task_id="bug_9281")
def fix_bug():
    client.chat.completions.create(...)  # tagged automatically
    run_subagent()  # nested calls too — no task_id parameter needed

with inferrail.track_task(task_id="..."): works the same way. Sync and async are both supported (attributed_async_http_client(base_url=...) for AsyncOpenAI/async frameworks); concurrent tasks never cross-contaminate. This is a small client-side convenience over the HTTP header above — no gateway or schema change, task_id only, no public API stability commitment yet. See docs/adr/0009.

Once tagged, inferrail report shows the all-up aggregate, while inferrail report --by <provider|model|route|attribute-name> aggregates receipts by any of these dimensions — customer, workflow, task_id, or anything else you've attached.

Referral early access

Referral access is opening soon. Planned early-access rewards are based on verified routed usage, not signup:

1 verified referral
→ +90 days of cost history for both sides

3 verified referrals
→ Pro for one year + unlimited seats

10 verified referrals
→ Founding Operator
→ permanent Pro
→ logo on the site
→ roadmap vote
→ private channel

25 verified referrals
→ Inferrail free for life
→ 20% recurring on additional teams referred

Program terms will be published when referral access opens.

See the current program presentation at tryinferrail.com.

How it works

InferenceEngine normalizes the request, resolves model to a route in inferrail.yaml (a pure config lookup — no cost/latency-aware selection in v0.1), calls the one provider adapter in this version (OpenAIProvider, generic over base_url — OpenAI itself, Azure OpenAI's compatible surface, vLLM, llama.cpp-server, or anything else speaking the same wire format), and emits a telemetry event and a receipt for every supported request, success or failure. Full lifecycle, package layout, and the streaming/retry boundaries: docs/ARCHITECTURE.md.

Privacy boundary

Inferrail's own local receipt, telemetry, and outcome records contain economic metadata and caller-supplied identifiers, not persisted prompts, responses, tool payloads, or free-form business outcome payloads. Structurally, the receipt and telemetry schemas have no field capable of holding message content, and test_inference_receipt_has_no_payload_fields enforces it. This is a claim about Inferrail's own local records, not about the request path as a whole — your configured provider still receives the real prompt either way; Inferrail is a pass-through gateway to it, not a privacy boundary against the provider.

Inferrail currently measures supported OpenAI chat-completions traffic. It is not a background monitor: it records while requests pass through the running process and serves nothing when that process is stopped. It does not enforce budgets or control provider spend.

inferrail try says this in its own output too, not just in the schema:

  Prompt stored     no
  Response stored   no

The full receipt shape, all fields:

{
  "receipt_id": "ir_1e6c916bac8940ca8a85",
  "route": "default",
  "provider": "openai",
  "model": "gpt-4o-mini",
  "status": "success",
  "prompt_tokens": 842,
  "completion_tokens": 191,
  "pricing": {
    "input_usd_per_million": "0.15",
    "output_usd_per_million": "0.60",
    "source": "https://developers.openai.com/api/docs/pricing",
    "verified_date": "2026-08-16"
  },
  "estimated_cost_usd": "0.000241",
  "attributes": { "customer": "acme", "workflow": "contract-review" },
  "total_latency_ms": 15.96,
  "retry_count": 0
}

If Inferrail can't verify a price for the (provider, model) pair, pricing and estimated_cost_usd are null — never a guessed or fabricated cost. You can check the no-payload claim yourself against a running gateway, not just take it on faith: docs/PRODUCT.md's verification walkthrough. Design rationale: docs/adr/0005.

MCP

pip install "inferrail[mcp]"

An MCP server (inferrail-mcp), published on the MCP registry as io.github.domondi1/inferrail, exposes Inferrail's local receipt ledger to any MCP-aware agent (Claude Code, Claude Desktop, Cursor, ...) as two read-only tools — neither executes inference nor spends provider budget:

{
  "mcpServers": {
    "inferrail": { "command": "inferrail-mcp" }
  }
}

Claude Code: claude mcp add inferrail -- inferrail-mcp. Full contract: inferrail-mcp/README.md.

Supported today

• POST /v1/chat/completions: streaming (stream: true, real SSE passthrough) and tool/function calling, single string message content, no n != 1
• GET /health
• One provider adapter, generic over any OpenAI-compatible HTTP endpoint
• Named-route + optional passthrough model routing (above)
• Per-route retry with backoff on transient provider errors
• Local structured telemetry and payload-free cost receipts for supported requests, plus inferrail report, grouped reports, and inferrail transaction <task-id>
• Customer-defined work_id attribution, append-only outcome declarations, and derived Work Economics via inferrail work outcome, inferrail work <work-id>, and inferrail work --all
• CLI: inferrail demo, try, serve (--quickstart), config check, report, transaction, work

Not yet

Honest edges, not silent gaps — full list in docs/PRODUCT.md:

• Cost- or latency-aware routing, or automatic failover to a different provider/model on error — routing is a static config lookup
• Budgets, spend limits, or blocking a request based on cost
• Any provider whose wire protocol isn't OpenAI-compatible (native Anthropic, Gemini, Bedrock, ...)
• The full OpenAI API surface — only /v1/chat/completions and /health exist; no embeddings, assistants, batch, images, or audio
• Multi-user auth or role-based access control — INFERRAIL_GATEWAY_TOKEN is one shared secret, not a user system
• Any hosted or cloud-operated component
• Non-LLM economic events (browser, search, compute/sandbox, MCP tool cost) in a TaskTransaction — its only event type today is inference
• Outcome or business-value linkage (success signal, revenue, margin) on a TaskTransaction — it aggregates cost only

Deployment boundary

Single node. The receipt ledger is a local append-only JSONL file, so every process that should appear in one report must write to one file on one filesystem.

• Concurrent writers to the same file are safe: each receipt is written as a single atomic O_APPEND write, so threads and multiple processes on the same host can share one ledger without interleaving or losing records.
• Not supported: several hosts writing to one ledger, aggregating ledgers across machines, or anything resembling a shared/hosted control plane. Running Inferrail on N hosts gives you N separate ledgers, and nothing in the product merges them.
• inferrail report and inferrail transaction read the whole file into memory. That is fine for the millions-of-bytes range a developer preview produces; it is not a query engine, and there is no retention, rotation, or compaction. Rotate the file yourself if it grows.

Anything beyond one host is out of scope for v0.x — see docs/PRODUCT.md.

Hosted services (optional, separate from the gateway)

Inferrail AP Exceptions (hosted/ap_exceptions/) is the optional hosted counterpart to the AP invoice-exception recovery SDK above: authenticated decision/persistence/reporting over HTTP, isolated per API key. It never executes a retry itself — that always happens in your own process. Not paid/x402-gated; a plain Authorization: Bearer <api-key> header. Full contract: hosted/ap_exceptions/README.md.

Paid capabilities (hosted, separate from the gateway)

Inferrail helps companies measure, attribute, and eventually govern the economics of work performed by AI agents. The gateway and receipts above are today's working part of that: privacy-preserving evidence of what AI work costs and what customer, workflow, or task its cost is attributed to. The two capabilities below extend that same foundation toward machine buyers. Both are experimental and Base Sepolia testnet only; neither controls external wallets, providers, or network spending.

Inferrail Work Economics is Inferrail's first hosted, paid capability: given caller-declared economic events for a unit of AI work, it returns a normalized cost receipt — known cost, a breakdown by resource class and supplier, and unit economics for the work — paid for over the x402 protocol by any agent with its own wallet — no Inferrail account required. Base Sepolia testnet only right now, not mainnet, not real money.

This is unrelated code, in hosted/, not part of the inferrail package — running the gateway above never requires it and never talks to it.

• Canonical endpoint: https://work.tryinferrail.com (manifest)
• Human-readable overview: tryinferrail.com/work-economics
• Full contract: docs/capabilities/work-economics.md
• Standalone buyer example: examples/work_economics_purchase.py

Inferrail Economic Authority (working name) explores voluntary coordination of a caller-declared spending boundary between agents. The boundary is caller-declared and the ledger is cooperative: Inferrail records and coordinates it entirely within its own service, and does not control any external wallet, provider, or network spending. This is not real-world spend enforcement. A buyer purchases a durable coordination boundary — a spending ceiling shared across agents, without double-allocating it — paid for over x402 by any agent with its own wallet. Base Sepolia testnet only. Whether session purchase is currently enabled on a given deployment is always authoritative from that deployment's own Agent Card, not this README.

• Canonical endpoint: https://authority.tryinferrail.com (Agent Card)
• Full contract: docs/capabilities/economic-authority.md
• Standalone client example: examples/economic_authority_session.py

Configuration

For a real deployment instead of quickstart defaults:

cp inferrail.example.yaml inferrail.yaml
cp .env.example .env      # then add a real OPENAI_API_KEY
inferrail config check    # validate without starting a server
inferrail serve

inferrail.yaml only ever holds the name of an environment variable for a secret, never the secret itself. Full shape (providers, routes, telemetry, receipts, pricing overrides): inferrail.example.yaml.

By default the gateway binds to 127.0.0.1:8000 with no auth. Set INFERRAIL_GATEWAY_TOKEN to require callers to send Authorization: Bearer <token> — see SECURITY.md.

Documentation

• docs/capabilities/ap-invoice-exception-recovery.md — AP invoice-exception recovery: full contract
• docs/PRODUCT.md — exact current scope
• docs/comparison.md — how Inferrail keeps prompt and response content out of its receipts
• docs/ARCHITECTURE.md — package layout, request lifecycle
• docs/adr/ — why specific structural decisions were made
• openapi.json / config.schema.json / llms.txt — machine-readable references for tooling and agents
• SECURITY.md

Development

git clone https://github.com/domondi1/inferrail.git && cd inferrail
pip install -e ".[dev,mcp,ap]"
ruff check . && mypy && pytest

pytest needs no API key or network access — see CONTRIBUTING.md. The ap extra is only needed to exercise OpenAIRetryAdapter's code path; the live-provider integration test still self-skips without OPENAI_API_KEY and INFERRAIL_LIVE_TESTS=1.

License

Apache License 2.0 — see LICENSE.