ai-tally (Python SDK)

The deep-context ingestion path for ai-tally. OpenTelemetry gen_ai.* native, with cost, feature-tag, identity, and agent extensions.

Core invariant: the SDK must never raise into the customer's code path. All internal errors are caught at the SDK boundary, recorded to self-observability, and the original call proceeds.

uv sync --extra dev
uv run ruff check .
uv run pytest

Zero runtime dependencies today (the schema + safety + sampling + guardrail primitives are pure-Python). OTel/OpenLLMetry integration lands in later tickets.

One-line connect (CTO-260)

The fastest path. aitally.init(key) needs no tenant_id (the ingest key is tenant-bound at the gateway), auto-instruments the official openai, anthropic and google-genai clients and Bedrock converse through boto3, boots the per-tenant HMAC key off-thread, and installs a background batching transport to /v1/batches.

import aitally

aitally.init("tally_sk_live_...")   # falls back to TALLY_KEY / TALLY_ENDPOINT env

# From here, unmodified provider calls are metered automatically - no record_* calls:
client.chat.completions.create(model="gpt-4o-mini", messages=[...])   # openai
client.messages.create(model="claude-sonnet-4-5", messages=[...])     # anthropic
client.models.generate_content(model="gemini-2.5-flash", contents=...)   # google-genai
bedrock.converse(modelId="us.amazon.nova-pro-v1:0", messages=[...])      # boto3 bedrock-runtime

init is idempotent, never blocks the calling thread, and never raises: a bad key, an unreachable gateway, or a missing provider library degrades to unattributed or disabled instrumentation with a one-time warning. None of these libraries is a dependency of the SDK; each is patched only if it is already installed.

The endpoint must be TLS (CTO-691, breaking)

Every request carries the ingest key as a Bearer token, and the HMAC bootstrap receives the tenant's raw key material, so the SDK only talks to its endpoint over TLS:

• https:// is always allowed.
• http:// is allowed without an opt-in only to a loopback host: localhost, a *.localhost name, 127.0.0.0/8 or [::1], so a local stack at http://localhost:8080 keeps working.
• Any other http:// host (a compose service name like http://gateway:8080, a LAN IP) needs an explicit opt-in: aitally.init(key, endpoint=..., allow_insecure_http=True), or TALLY_ALLOW_INSECURE_HTTP=1 in the environment. With it the SDK works and logs one warning naming the host (never the key). An explicit allow_insecure_http=False wins over the env var.
• Any other scheme, a URL with no host, and a URL carrying userinfo (user:pw@), a query or a fragment are refused even with the opt-in.

The host is compared after parsing the URL, so http://localhost.evil.com and http://127.0.0.1@evil.com are not loopback, and a host with whitespace, control characters or percent-escapes is refused. Loopback requests bypass http_proxy, so a configured proxy never sees the key in cleartext. A *.localhost name is trusted by name: most resolvers answer it locally (RFC 6761), but one that appends a search domain may not, so prefer localhost or 127.0.0.1 if yours does. hash_account() called before aitally.init() has finished its bootstrap fetches on its own and reads only TALLY_ALLOW_INSECURE_HTTP, not the init() argument.

A refused endpoint behaves like a missing key: aitally.init() logs one warning and disables telemetry rather than raising, and no request is sent. The check sits where every request URL is built, so fetch_hmac_key, a directly built BatchingTransport (which raises InsecureEndpointError, a ValueError), hash_account and the tally CLI (which exits 1 with the reason) all follow the same rule. The SDK also no longer follows HTTP redirects from the endpoint, since urllib would carry the key to the redirect target: a batch answered with a 3xx is dropped and counted as rejected (rejected_span_count), not retried, and an HMAC bootstrap answered with one fails and leaves accounts unattributed. Point endpoint at the gateway's final URL.

This is a breaking change: before it, any http:// endpoint was accepted silently. The SDK is unpublished at 0.0.1, so it lands without a version bump.

What aitally.init wraps (CTO-260, CTO-558)

Not wrapped, so still a aitally.record_llm_call after the call: Bedrock invoke_model and invoke_model_with_response_stream (their usage is in a model-specific body, or in undocumented headers with no cache-read count, so a figure from them could silently undercount), the legacy google-generativeai package, and any other client library.

• Provider name for OpenAI-compatible hosts. The openai library also talks to Fireworks, Groq, Gemini's OpenAI-compatible endpoint and others, so each call is labeled from the client's base_url host, not assumed to be OpenAI: api.openai.com is openai, api.fireworks.ai is fireworks-ai, generativelanguage.googleapis.com is google, and Groq, Together, DeepSeek, xAI, OpenRouter, Mistral and Azure get their own names (their calls show a blank cost until the catalog prices them, never OpenAI's rates). A host that matches nothing is recorded as openai-compatible, which no catalog row prices. Only the host is read; the URL never reaches the span. OpenAI's own data-residency hosts (eu.api.openai.com, us.api.openai.com) are openai.

• Gemini tokens. prompt_token_count plus tool_use_prompt_token_count is input, candidates_token_count plus thoughts_token_count (thinking, billed as output) is output, and cached_content_token_count is the cached share of input. With automatic function calling, each round trip is its own span, because each is billed.

• Bedrock model ids. us.amazon.nova-pro-v1:0 and foundation-model ARNs are recorded as amazon.nova-pro, the catalog's key. An ARN whose model cannot be read (an application inference profile, a provisioned model) records no model and no cost rather than the ARN. Bedrock's inputTokens excludes cached tokens, so both cache buckets are added back into input. A converse_stream span is emitted once the caller has read the stream to its end.

• Streaming tokens. OpenAI reports usage only when stream_options={"include_usage": True} is set. By default a stream without it emits a span with null token counts (honest blank, never a fabricated zero). Pass aitally.init(..., instrument_stream_usage=True) to have the OpenAI wrapper add include_usage when the caller did not, so streamed calls price fully.

• Accounts. Set the customer once with with_account("acct_..."); every auto-instrumented span in the scope carries the HMAC'd account hash, computed in-process under the bootstrapped tenant key. Until the bootstrap completes (or if it fails), accounts land unattributed - never a raw id.

• What is automatic vs. app-side. The one-liner captures LLM provider calls only. Vector search (aitally.record_vector_call), your own tool calls (aitally.record_tool_call), and embeddings not made through the patched client (aitally.record_embedding_call) remain explicit one-liners that delegate to the process-global client. These are safe no-ops before init.

• Lifecycle. aitally.flush() drains buffered spans (also drained at atexit); aitally.uninstrument() reverses all patches and tears the client down (used by tests).

• Choosing which clients are patched. instrument=True (the default) patches every library in the table above, instrument=False patches none, and a collection of names patches only those: "openai" (which also covers OpenAI-compatible hosts reached through the openai library, Fireworks included), "anthropic", "google" (google-genai; "gemini" works too) and "bedrock". For example aitally.init(key, instrument={"openai", "anthropic"}).

• Clients pointed at the ai-tally proxy are left alone. The proxy already records those calls, so a client whose base URL is an ai-tally proxy route (*.proxy.ai-tally.com, or ingest.ai-tally.com under /openai, /anthropic or /gemini) or that sends the proxy's X-Tenant-Key header is passed straight through rather than metered a second time. The header check is what covers a self-hosted proxy on your own domain.

Upgrading: remove manual lines for Gemini and Bedrock Converse

Earlier versions did not wrap google-genai or Bedrock converse, so the setup guidance (and the onboarding agent's recipe) had you add a aitally.record_llm_call(...) after those calls. aitally.init() now records them automatically. Delete those manual lines. To keep metering a provider by hand instead, leave it out of instrument: instrument={"openai", "anthropic"} keeps Gemini and Bedrock manual.

Until you do, a safety net keeps the count right. When a record_llm_call matches a call the SDK recorded automatically moments earlier (same provider, model and token counts, on the same client, in the same thread or asyncio task, within a few seconds), the manual copy is not sent, and a one-time warning names the fix. Two identical calls with two manual lines still count as two calls, and a manual call with no automatic counterpart is always sent. The net only sees a manual line that runs in the same thread or task as the call it duplicates, so treat it as a net, not a fix.

Hashing an account for the proxy path

The zero-code proxy holds no HMAC key. To send a pre-hashed X-Tally-Account-Id-Hash, compute it on your own machine with the same key the SDK uses:

from aitally import hash_account
h = hash_account("acct_northwind")          # uses the bootstrapped tenant key

python -m aitally.hash_account acct_northwind  # CLI form, reads TALLY_KEY / TALLY_ENDPOINT

The gateway endpoint GET /v1/tenant/hmac-key that the bootstrap fetches is delivered in a separate PR; the SDK codes against its contract (spec §3.2).

Command line (CTO-867)

The SDK installs a tally command. It needs no extra and uses the same key and endpoint as aitally.init().

tally login            # reads a key without echoing it; stores it 0600 in ~/.config/tally/config.ini
tally health           # the key's scope and the last 24 h of metered model calls
tally keys             # the organization's keys, metadata only (a write or admin key)
tally test-event       # sends one span with no model, tokens, cost or body; prints what was accepted
tally spend --month 2026-10   # the month's metered spend by feature and provider

TALLY_KEY wins over the stored key, so CI needs no config file; TALLY_CONFIG moves the file; --endpoint or TALLY_ENDPOINT points it at a self-hosted gateway, under the same TLS rule as aitally.init() (TALLY_ALLOW_INSECURE_HTTP=1 for a non-loopback http:// host). The key is never printed, not even in part. A figure the gateway could not read prints as unknown with the reason, never 0, and a month with unpriced model calls prints its total as "at least". Every read command takes --json.

Local replay samples (CTO-1110)

For teams whose request text must not leave their servers, the SDK can keep replay samples on the machine that already holds them instead of sending them to Tally:

aitally.init(replay_capture="local", replay_dir="/var/lib/tally/replay")

It captures only while an admin has turned on local capture for the organization (a switch of its own, separate from hosted replay), samples 1% of non-streamed calls (replay_sample_rate), and keeps 7 days (replay_retention_days). Nothing is sent: no provider key, no headers, and no body leaves the box. The files hold real prompts and answers, so your own data rules apply to that folder.

Files are <replay_dir>/<UTC day>/<host>-<pid>.jsonl.gz (a +<n> suffix after a failed write), with a .tally-replay marker at the top. Tally only ever deletes YYYY-MM-DD/*.jsonl.gz files in a folder carrying that marker, and never follows a symlink. When local capture is turned off for the organization, the SDK deletes every sample file under replay_dir. To delete them yourself:

tally eval purge --samples /var/lib/tally/replay                     # everything
tally eval purge --samples /var/lib/tally/replay --before 2026-10-01 # only older days

aitally.purge_local_samples(replay_dir, before=...) does the same from code.

Running an eval on your own server

pip install openai anthropic        # whichever providers your candidates and judge use
tally eval --samples /var/lib/tally/replay --models gpt-4o-mini,claude-haiku-5-5 --max-spend-usd 5

Each sample is replayed on every candidate with your own provider keys (OPENAI_API_KEY, ANTHROPIC_API_KEY, read by the provider libraries, never by Tally). A judge then compares each candidate's answer with the answer your product actually gave, using the same rubric as Tally's hosted Compare. You get one row per candidate: replay cost, judge cost, win rate against the original answer (with a 95% interval), refusal rate, and how many samples were scored, skipped and failed. Skipped and failed samples never count as losses. A figure that can't be known prints as unknown, never $0.00.

• The judge is your organization's judge model unless you pass --judge (or aitally.init(eval_judge_model=...)).
• --max-spend-usd covers candidate and judge calls together. Every call reserves its worst case before it is made, and no call is made that could pass the cap. The command prints the worst case and asks before it starts; --yes skips the question in CI.
• Every call is sent with an output cap: --max-output-tokens and --judge-max-output-tokens, 1024 by default. An answer cut off at the cap counts as failed.
• --format json also checks that each answer is valid JSON. --json prints the table as JSON.
• --report sends the numbers (model names, counts, costs, rubric version, no prompts or answers) to Tally, where the run shows on Compare as run on your server.

tally eval refuses to run while local capture is turned off for the organization. If an admin turned it off, it also deletes the samples. aitally.run_local_eval(...) does the same from code.

Tagging spend with a customer account

An account_id says which of your customers a call belongs to. It is what turns a cost total into cost per customer.

The id is context-scoped: a web app knows the customer once, at request start, so you set it once and every span inside the scope picks it up. Requiring it on every call would be noise, and noise gets skipped.

from aitally.client import TallyClient
from aitally.context import start_trace, with_account
from aitally.hmac_keys import HmacKeyRegistry

registry = HmacKeyRegistry()
registry.provision("tenant-a")
client = TallyClient(tenant_id="tenant-a", hmac_registry=registry)

# In a request middleware: resolve the customer once, wrap the handler.
with start_trace(feature_tag="support-bot"), with_account("acct_northwind"):
    client.record_llm_call(provider="openai", model="gpt-4o", usage=usage)
    client.record_tool_call(provider="openai", tool="web_search")
    # ...every span emitted inside this block carries the same account.

One call can override the scope, which is what a batch job that walks several customers needs:

with start_trace():
    for account in accounts:
        client.record_llm_call(
            provider="openai", model="gpt-4o", usage=usage, account_id=account
        )

with_account(None) clears the account for a block. That is the opt-out for a background task that must not inherit its caller's customer.

What actually goes on the wire

The raw account_id never leaves your process. It is HMAC-SHA256'd under your per-tenant key at emit time, exactly like a user id, and the span carries only:

Because the key is per tenant, the same account id hashes differently for two different tenants, so an account cannot be correlated across them.

If the client has no tenant_id or no hmac_registry, the span is emitted unattributed with a one-time warning. It is never dropped and the raw id is never substituted.

The optional label

with with_account("acct_northwind", label="Northwind Traders"):
    ...

The label is wire-only. The gateway upserts it into a label store keyed on the account hash and does not write it to the span row, so no customer name lands in the telemetry store. Labels are optional per account: set none and the dashboard falls back to a shortened hash, which is a supported way to run. A label with no account id alongside it is dropped, since there is nothing to key it on.

Onboarding MCP server (CTO-261)

aitally.init() meters the LLM layer on its own, but the other layers need app code: the vector, tool and embedding record_* calls, and the per-customer attribution only your app can resolve. The onboarding MCP server hands your own coding agent the maintained recipes and generated snippets for that wiring. Your source never leaves your machine: the server holds only the recipe catalog and the SDK surface, and it returns code, never an applied edit.

Install the SDK with the mcp extra (it is optional, the SDK runtime itself has no dependencies). The SDK is not on PyPI yet, so install it from the repository:

pip install "ai-tally[mcp] @ git+https://github.com/jain-aanchal/ai-tally#subdirectory=sdk/python"

That installs the tally-onboarding-mcp console script, which speaks MCP over stdio.

Every example below passes your ingest key as TALLY_KEY (the same tally_sk_... key aitally.init() uses, from the dashboard's Setup page). Only coverage_report uses it, to check which layers are reaching ai-tally; the other tools work offline without it.

Claude Code

claude mcp add ai-tally-onboarding --env TALLY_KEY=tally_sk_... -- tally-onboarding-mcp

Or add it to .mcp.json in your project root so your team picks it up too. Keep the key out of the committed file: ${TALLY_KEY} is expanded from each developer's own environment.

{
  "mcpServers": {
    "ai-tally-onboarding": {
      "command": "tally-onboarding-mcp",
      "args": [],
      "env": { "TALLY_KEY": "${TALLY_KEY}" }
    }
  }
}

Cursor

Add this stanza to ~/.cursor/mcp.json (global) with your key in env. A project-level .cursor/mcp.json works too, but a file holding a key must never be committed:

{
  "mcpServers": {
    "ai-tally-onboarding": {
      "command": "tally-onboarding-mcp",
      "args": [],
      "env": { "TALLY_KEY": "tally_sk_..." }
    }
  }
}

If tally-onboarding-mcp is not on your agent's PATH, point command at the interpreter that has it installed instead: "command": "/path/to/.venv/bin/tally-onboarding-mcp".

The tools

Configuring coverage_report

coverage_report is the only tool that talks to anything outside your machine. It calls the gateway's coverage probe, which is the side that can read your telemetry.

For a customer, one variable is enough:

Operators running the tool against a tenant they do not hold a key for use the service-token path instead. The service token is used only when TALLY_KEY is unset:

TALLY_MCP_TENANT_ID is deliberately not the retired TALLY_TENANT_ID, which used to scope the dashboard and which some older deploy manifests still set, inertly, to local-dev. The tool never reads the retired name, not even as a fallback: a stale value would silently scope your coverage report to the wrong tenant. If it finds TALLY_TENANT_ID set while TALLY_MCP_TENANT_ID is not, it answers probe_available: false and tells you to rename the variable, rather than guessing with it. A value that is not a canonical UUID is refused the same way, because the probe binds it into a UUID read filter where a tenant name matches nothing and would look like an honest empty answer.

The key or token is held by reference: the tool reads the variable at the moment it calls the probe and never stores, echoes or logs the value. The tenant_key argument is likewise reported only as present or absent, never sent onward.

The token is also never carried off the origin you configured: a redirect to a different scheme, host or port is refused rather than followed, so a load balancer that 301s elsewhere cannot be handed your service token.

Leave these unset and the tool answers probe_available: false with every layer unknown and the reason attached. That is also what you get if TALLY_GATEWAY_URL is rejected as unsafe, or if the gateway is unreachable, answers an error, times out, redirects off-origin, or returns something unreadable or oversized. The result shape is the same on both paths, so reason is always present (empty when the probe answered). None of those ever turns into "this layer is not wired": a layer is reported covered only when the probe returns a span count above zero to prove it, re-derived from that count rather than taken on the wire's word, so a payload claiming coverage with no evidence is downgraded back to unknown.

Pass wired (the layers you just instrumented) to separate "wired, awaiting first event" from "not wired" on a dark layer. It only softens the wording; it can never produce coverage.

Two things the server will not do. It never guesses which customer a request belongs to: pass generate_middleware the header or resolver you confirmed, and with no answer it returns a reported gap and the account layer stays unattributed. And it never invents an SDK call: a stack with no recipe comes back as a gap, and every hole it cannot fill from the call site you passed is left as a visible <FILL:...> marker for you to complete.

One thing the server will not let you do twice. aitally.init() already patches the openai, anthropic and google-genai clients and Bedrock converse in your process, so those call sites need no edit. detect_stack reports any such provider under already_covered, and the manual llm.generic.call recipe deliberately does not match a patched call shape: adding record_llm_call beside one would meter the same call twice and double your reported cost. Use that recipe only where the patch does not reach, such as a raw /v1/chat/completions POST, Bedrock invoke_model, ollama, or a self-hosted or gateway-fronted model.

Both mcp 1.x (FastMCP, 1.2.0 and up) and mcp 2.x (MCPServer) are supported. Without the extra installed, launching the server fails with a clear error rather than starting a server that does nothing. The extra is also what makes the tools usable at all: onboarding_mcp ships in the base wheel so the console script resolves, but the recipe catalog is YAML and pyyaml comes with the extra, since the SDK runtime itself stays dependency-free.

Node and TypeScript apps (CTO-556)

The SDK is Python, so for a Node or TypeScript app the agent wires the two paths that need no SDK, and the server still runs from this Python package on your machine. Pass detect_stack your package.json on its own (both dependencies and devDependencies are read) and it answers language: "node" with only Node recipes; a Python manifest never gets them, and a package.json never gets Python SDK code.

<ingest> is your deployment's ingest URL, https://ingest.ai-tally.com on the hosted service. The proxy URLs are the ones the dashboard's connect snippets show, and a test holds the two together. On the proxy path your provider key is forwarded to the provider unchanged and is not recorded, but it does reach AI Tally in transit; the OTel path never sends it. The proxy answers 403 until an organization admin turns it on under Settings > API keys.

Per-customer attribution is the same floor as in Python: the agent asks how your app knows the customer and never guesses. The proxy holds no HMAC key and will not hash a raw id, so the account recipe computes the HMAC-SHA256 digest in your process under the tenant key from /v1/tenant/hmac-key (the same digest the Python SDK produces) and sends only that. A raw customer id is never sent. Without the header the spend lands unattributed, reported as a blank rather than a guessed customer.

If a Node app has both a proxied client and an OpenTelemetry setup that emits gen_ai.* spans, detect_stack flags it under already_covered: pick one path per call, or the call is metered twice. Providers with no proxy recipe yet (Gemini, Bedrock, Mistral, Cohere, Groq), a Vercel AI SDK app with no OpenAI or Anthropic provider package, and Node vector databases come back as named gaps. instrument_call_site, generate_middleware and explain_layer stay Python-only, since they emit or explain Python SDK calls; for a Node framework such as express, generate_middleware answers with a gap that points at account.node.header_hash. The hosted PR bot below still reads Python repos only.

Hosted repo PR bot (CTO-261)

The MCP server above hands the recipes to your own coding agent. The PR bot is the other end of the same catalog: give it scoped access to a repo and it runs the loop server-side and opens a reviewed pull request. Same recipes, same refusals, no coding agent needed on your side. It is a headless bot rather than a GitHub App, so the access you grant is a token you hold and can take back.

export TALLY_ONBOARDING_GITHUB_TOKEN=github_pat_...
tally-onboarding-bot \
  --repo acme/widgets \
  --account-source 'request.headers.get("X-Customer-Id")' \
  --feature-tag support-bot

Omit --account-source and the bot does not pick one for you: it opens the PR carrying the account question and the candidate resolvers it found, leaves the account layer unattributed, and says so in the body. --dry-run proposes the diff and stops before creating a branch.

Supplying and revoking the token

1. In GitHub, go to Settings, Developer settings, Personal access tokens, Fine-grained tokens, and generate a token whose repository access is only the repo you want the PR in.
2. Give it exactly two repository permissions: Contents: read and write (to push the new branch) and Pull requests: read and write (to open the PR). Nothing else is used. Set the shortest expiry you can live with.
3. Put the value in the environment variable named by --token-env (default TALLY_ONBOARDING_GITHUB_TOKEN), or in your secret manager and inject it from there. The bot holds the variable NAME, never the value: it reads it at the moment git or the API needs it, hands it to git through GIT_ASKPASS so it never lands in .git/config or in a process listing, and redacts it from anything it prints.
4. To revoke, delete the token on that same page, or unset the variable. There is no installation to uninstall and no stored copy to clean up, which is the point of a token you hold rather than an app you grant.

A GitHub App is the harder-edged version of this (short-lived installation tokens, grants managed in GitHub) and stays the hardening path. It would change how the credential is minted, not what the bot is allowed to do.

What it refuses to do

Enforced in onboarding_bot/guards.py and covered by tests/test_onboarding_bot_guards.py, not merely promised here:

• It never pushes to a default branch. Every push resolves the repo's default branch first and refuses it, along with main / master / trunk / develop whatever the remote reports. It pushes the one new branch it created, and force pushes are refused.
• It never merges. Git subcommands and GitHub endpoints are both allowlisted; merge (and rebase, cherry-pick, reset) is on neither, so no code path reaches one. The PR waits for a human.
• It keeps no source. The clone is shallow, lives in a temporary directory, and is deleted in a finally on every exit path including a refused run, and by a SIGTERM / SIGINT handler so a runner that stops the job does not leave the clone behind either. A SIGKILL cannot be caught by anything, so the promise is every exit path the process controls. A cleanup that fails is reported in cleanup_error, never swallowed. What the run returns is paths, counts, generated code and gaps, never your source.
• It invents nothing. Every line it writes comes from the recipe catalog. A value it cannot derive from the call site stays a visible <FILL:...> hole, and a block with a hole is inserted commented out so nothing runs on a guessed value.
• It claims nothing it did not do. layers_wired (in the JSON result, the commit subject and the PR table) lists only blocks that actually meter. A block inserted commented out is reported separately as inactive.
• It never commits code that does not compile. Every block goes in at a statement boundary resolved from the parsed file, and every patched Python file is run through compile() before anything is written. A file it cannot patch cleanly is left untouched and reported as a gap.

Each run gets its own branch (tally/onboarding/<run id>); pass --branch-suffix to name it yourself. If the branch is already on the remote the run stops and says so instead of failing at push. If the push succeeds but the PR call does not, the exit is non-zero and the printed result carries orphan_branch so you can find the branch it left.