bir-sdk 0.2.0

Minimal local tracing SDK for LLM applications.

Bir Python SDK

Minimal, zero-runtime-dependency, local-first tracing and evals for Python LLM applications.

Bir records traces, spans, generations, tool calls, retrievals, and scores to local JSONL without requiring a server. Start locally, evaluate deterministic regressions, and send events to a Bir server when you want to inspect them in a dashboard.

Installation

python -m pip install bir-sdk

The distribution name is bir-sdk; the import name is bir. Runtime installation has no third-party dependencies. Bir also ships inline type annotations and a PEP 561 py.typed marker.

An opt-in otel extra (pip install 'bir-sdk[otel]') adds an OpenTelemetry/OTLP exporter that forwards recorded traces to an existing observability backend; the runtime install stays dependency-free without it. See Forwarding traces to OpenTelemetry.

Quickstart

from bir import generation, observe, score


@observe()
def answer_question(question: str) -> str:
    with generation("local.llm", model="demo-model") as gen:
        response = f"Answer: {question}"
        gen.set_output(response)
        gen.set_usage(input_tokens=12, output_tokens=24)
    score("helpfulness", 0.82)
    return response

Events are written to .bir/traces.jsonl by default. Input and output capture is disabled unless you explicitly enable it.

Inspect them from the command line without a server: bir traces lists recorded traces, bir show <trace-id> prints one trace as an indented tree of its spans, generations, tool calls, and scores, and bir stats summarizes trace counts, token usage, cost per currency, and latency (count, mean, and p95) for a quick cost or health check (add --json to any of them for a structured form). See CLI & environment.

When capture is on, Bir redacts common secret-like fields and text before anything is written. Those built-in rules always apply and cannot be turned off, but you can widen them for your own credential names and formats with configure(additional_secret_keys=[...], additional_redaction_patterns=[...]). See capture and privacy.

Correlating your logs with traces

Read the active ids with get_current_trace_id() and get_current_span_id() to stamp your application's own logs and metrics, so they line up with Bir traces later. Both return None outside any trace and never raise:

import logging

from bir import get_current_span_id, get_current_trace_id, observe


@observe()
def answer(question: str) -> str:
    logging.info(
        "handling question",
        extra={"trace_id": get_current_trace_id(), "span_id": get_current_span_id()},
    )
    return "ok"

get_current_trace_id() returns the active trace root id and get_current_span_id() the innermost open span, generation, or tool call (the trace root when none is open). The values match the trace_id/parent_id later written to the JSONL, and each asyncio task and thread sees its own ids. The accessors are read-only — there is no setter and no context is exposed for injection or cross-process propagation.

Attaching metadata discovered mid-body

Every trace-work context manager — trace(), span(), generation(), tool_call(), and retrieval() — exposes set_metadata(...) so you can record context that only becomes known while the body runs (a resolved route, a cache-hit flag, a request id) before the event is written:

from bir import generation, observe


@observe()
def answer(question: str) -> str:
    with generation("local.llm", model="demo-model", metadata={"provider": "demo"}) as gen:
        response = f"Answer: {question}"
        gen.set_metadata({"route": "fast", "cache_hit": False})
        gen.set_output(response)
    return response

set_metadata() merges into any metadata passed at creation time — later keys win, including across repeated calls — and the merged metadata is redacted before it is written with the same rules as captured input and output, so secret-like fields never reach the JSONL. It works with both with and async with, and the argument must be a mapping.

Estimating cost from a local price table

Cost is user-provided by default — Bir bundles no prices because provider prices go stale. If you would rather not call set_cost() on every call, supply your own per-token rates once with configure(model_prices=...) and Bir fills the cost from token usage:

from bir import configure, generation, observe

configure(
    model_prices={
        "gpt-4o-mini": {"input": 0.00000015, "output": 0.0000006},
        "mistral-large": {"input": 0.000002, "output": 0.000006, "currency": "EUR"},
    }
)


@observe()
def answer(question: str) -> str:
    with generation("chat", model="gpt-4o-mini") as gen:
        gen.set_usage(input_tokens=1000, output_tokens=400)
        # No set_cost(): input_cost/output_cost/total_cost are derived from the rates.
    return "ok"

Each entry sets a non-negative input and/or output per-token rate plus an optional currency (default USD). The cost is derived only for a generation that has the matching token counts and no explicit set_cost() — an explicit cost always wins, and a usage without the needed token split is left without a cost. The table is validated at configure() time, ships no bundled prices, and keeping the rates current is your responsibility. With no table configured, cost behavior is unchanged. See core API.

Tracing generators and streaming

@observe() also traces generator and async-generator functions across their full iteration, not just their creation. The wrapper stays lazy — the body does not run and nothing is written until the first iteration — and the trace stays open from the first next()/await __anext__() through exhaustion, so spans and generations created in the body (for example while consuming a streamed LLM response) attach to it:

from bir import generation, observe


@observe()
def stream_answer(question: str):
    with generation("local.llm", model="demo-model") as gen:
        chunks = []
        for token in ("Ans", "wer", "!"):
            chunks.append(token)
            yield token
        gen.set_output("".join(chunks))

The trace is finalized when the generator is exhausted (a successful trace), raises (a redacted error, re-raised unchanged to the consumer), or is closed or cancelled early. An early close()/aclose() or a cancellation is recorded as a successful trace whose metadata.generator.outcome is "closed", and it resets all trace context so nothing leaks into later work. send/throw/close (and asend/athrow/aclose) and the body's finally blocks all behave exactly as they would without the decorator, and concurrent async generators running in separate tasks stay isolated. Yielded values are never buffered; with output capture enabled only a bounded yielded-item count is recorded under metadata.generator.items.

The optional provider integrations ship async counterparts (trace_chat_completion_async, trace_messages_async, trace_completion_async, and so on) for async clients such as AsyncOpenAI, AsyncAnthropic, and litellm.acompletion. Each awaits the provider coroutine inside an active trace and records one generation; the streaming wrappers resolve to an async iterator you consume with async for, never buffering the stream. The synchronous wrappers likewise accept stream=True — yielding the provider's chunks unchanged and recording the accumulated text and final token usage once the stream is consumed — across OpenAI (Chat Completions and Responses), Anthropic, Gemini, Mistral, Cohere, LiteLLM, and Vertex AI. AWS Bedrock's Converse stream is a distinct method rather than a stream=True flag, so it has its own trace_converse_stream wrapper that yields the stream's events unchanged and records the same way.

For agent frameworks, Bir ships dependency-free callback handlers that map a framework's own events into Bir traces without importing the framework: BirCallbackHandler for LangChain, BirLlamaIndexHandler for LlamaIndex, and BirAgentsTracingProcessor for the OpenAI Agents SDK. The Agents processor implements the SDK's tracing-processor interface, turning an agent run into a Bir trace whose model spans become generations and tool spans become tool calls; register it with agents.add_trace_processor(BirAgentsTracingProcessor()).

Local persistence and concurrency

Trace appends and size-based rotation are serialized across threads and local processes that write the same trace path. Opt-in sent-ID bookkeeping uses a separate lock around sidecar merge and replacement, so concurrent workers and bir send processes preserve the union of accepted IDs. The implementation is stdlib only: it uses flock on POSIX and byte-range locking on Windows, with stable hidden lock files beside the trace and sidecar files.

These locks are advisory, so every writer must use Bir's persistence path. Cross-host coordination and filesystems that do not implement normal local advisory-lock semantics are not supported; use one local trace path per host in those deployments. Lock files may remain on disk and must not be deleted while Bir processes are active.

By default send_events() and bir send upload only the active trace file. Pass include_rotated=True (or bir send --include-rotated) to also upload retained size-rotated files oldest-first, deduplicated by event ID, so rotation does not strand unsent events. Both send_events()/bir send and send_experiment()/bir send-experiment retry transient failures (network errors, timeouts, and HTTP 5xx) with bounded exponential backoff via retries and backoff, while HTTP 4xx and malformed inputs fail immediately. See server uploads.

Forwarding traces to OpenTelemetry

If you already run an OpenTelemetry backend, you can replay locally recorded Bir traces as OpenTelemetry spans and ship them over OTLP. This is opt-in and never runs on its own: nothing imports opentelemetry until you call the exporter, and it only reads your local JSONL — it never writes to or alters it.

Install the extra, then forward loaded traces:

python -m pip install 'bir-sdk[otel]'

from bir import load_traces
from bir.integrations.otel import export_traces_to_otlp

export_traces_to_otlp(
    load_traces(),
    endpoint="http://localhost:4318/v1/traces",
    service_name="rag-api",
)

export_traces_to_otlp() also accepts a single LoadedTrace, an iterable of them, or a path to a trace file (loaded via load_traces). Each Bir trace becomes one OpenTelemetry trace: the trace root maps to a root span and every other event maps to a child span linked by parent_id, carrying over start/end times and success/error status. Attributes follow the GenAI semantic conventions where they exist (gen_ai.request.model, gen_ai.usage.input_tokens / gen_ai.usage.output_tokens) with bir.* attributes for the rest (event type, score value, token totals, cost, and the originating Bir ids). Pass headers= for backend auth, or inject your own configured span_exporter= for a different transport. Calling the exporter without the extra installed raises a clear error pointing you to pip install 'bir-sdk[otel]'.

Evaluations and experiments

Bir ships deterministic, local-first evaluators and an experiment runner that scores a task over a dataset and persists per-example results and a summary under .bir/experiments/. Use run_experiment() for synchronous tasks, or run_experiment_async() when your task is a coroutine such as an async provider client:

import asyncio

from bir.evals import Dataset, DatasetExample, contains, run_experiment_async


async def answer(question: str) -> str:
    ...  # await your async model client


result = asyncio.run(
    run_experiment_async(
        "prompt-v1",
        dataset=Dataset([DatasetExample(id="q1", input={"question": "Hi"})]),
        task=answer,
        evaluators=[contains("Hi")],
        max_concurrency=8,
    )
)

run_experiment_async() runs up to max_concurrency examples concurrently while keeping results, JSONL rows, and summary aggregates in dataset order. It accepts async tasks, plain sync callables, and sync callables that return an awaitable, and otherwise matches run_experiment(). See local evals and experiments.

Compare a candidate against a baseline and gate CI on regressions with compare_experiments() or bir eval-gate. A global tolerance bounds how far a shared evaluator may drop, score_tolerances (and repeatable --score-tolerance NAME=VALUE) override that per evaluator, and missing_score (--missing-score {ignore,regress}) decides whether an evaluator dropped from the candidate fails the gate:

bir eval-gate baseline.jsonl candidate.jsonl \
  --tolerance 0.01 --score-tolerance latency_under=0.05 --missing-score regress

The command exits 1 exactly when the policy reports a regression and prints a machine-readable diff with effective_tolerances, missing_score, and regression_reasons.

Documentation

The documentation site covers the quickstart, core API, capture and privacy, server uploads, optional integrations, and local evals and experiments.

Build it locally with the isolated documentation extra:

python -m pip install -e ".[docs]"
mkdocs build --strict

CI installs the same docs extra and runs the strict build once on every pull request and push to main, so invalid navigation, links reported by MkDocs, and build warnings block the change.

For local SDK development, install .[dev] and see the release checklist.

python -m pip install -e ".[dev]" pyright
pyright
python scripts/verify_release.py

Release verification builds the wheel without network access from the complete bir package tree, checks its contents and RECORD hashes, then installs it into a clean virtual environment. The installed-wheel smoke test imports bir.evals, bir.cli, and every optional integration module without installing provider SDKs.

The checked example tests use only standard-library test utilities, so Pyright's release gate is hermetic whether tooling is installed in a repository .venv or in CI's active interpreter. Pytest remains optional development tooling.

License

Bir is licensed under the Apache License 2.0.