bir-sdk 0.1.0

Minimal local tracing SDK for LLM applications.

Bir Python SDK

Minimal local tracing SDK for Python LLM applications.

Bir records traces, spans, generations, tool calls, and scores to local JSONL without requiring a server. Start locally, then send events to the Bir FastAPI server when you want to inspect them in the dashboard.

Installation

After the first package release:

python -m pip install bir-sdk

The distribution is published on PyPI as bir-sdk; the import name is bir (e.g. from bir import observe).

For local development from this repository:

python -m pip install -e ".[dev]"

Quickstart

from bir import generation, observe, retrieval, score, span


@observe()
def answer_question(question: str) -> str:
    with span("retrieve_context"):
        with retrieval("search_docs", query=question) as result:
            result.add_document(id="doc-1", text="local context")
            documents = ["local context"]

    with generation("local.llm", model="demo-model") as gen:
        response = f"{documents[0]}: {question}"
        gen.set_output(response)
        gen.set_usage(input_tokens=12, output_tokens=24)
        gen.set_cost(input_cost=0.000012, output_cost=0.000048)

    score("helpfulness", 0.82)
    return response

Trace, span, tool call, generation, and score events are written as JSONL to:

.bir/traces.jsonl

Writes to the local trace file are serialized within the SDK process so multi-threaded sync applications keep the JSONL file line-delimited and parseable.

Manual Trace Contexts

Use trace() when your workflow is easier to wrap with a context manager than a decorator:

from bir import generation, score, span, trace

with trace("answer_question", metadata={"kind": "manual"}):
    with span("draft_answer"):
        with generation("local.llm", model="demo-model") as gen:
            response = "ok"
            gen.set_output(response)
    score("helpfulness", 0.82)

Read Local Traces

You can also read local traces back from the same file:

from bir import load_traces

for trace in load_traces():
    print(trace.name, trace.status, trace.duration_ms)
    for event in trace.events:
        print(event.type, event.name)

Send Events To The Server

To send local events to a running Bir server:

from bir import send_events

result = send_events("http://127.0.0.1:8000")
print(result.accepted, result.attempted, result.skipped)

send_events() posts local JSONL events to the Bir server, batching them into a single request when the server supports it and otherwise posting them to /v1/events one at a time. It uses the Python standard library, reports how many local events were attempted, how many were newly accepted, and how many were skipped by an idempotent server response, raises RuntimeError when the server rejects an event or cannot be reached, and does not remove local events after sending. Re-sending the same file is safe against the Bir server because duplicate event IDs are treated as already ingested. Complete traces are sent root-first so the server receives the trace event before its spans, tool calls, generations, and scores.

Privacy And Capture

Input and output capture is disabled by default. Enable it globally with configure() or for a single function with @observe(capture_inputs=True, capture_outputs=True). Common secret-like fields such as api_key, authorization, password, secret, and token are redacted before events are written. Captured strings, fallback object representations, and captured error messages are also scanned for common secret-like text patterns before events are written. Redaction is best-effort, so keep capture opt-in for sensitive payloads and review what your application records.

from bir import configure

configure(capture_inputs=True, capture_outputs=True)

Captured values are normalized to JSON-compatible data before writing. Non-finite floats such as NaN and Infinity are stored as strings, and deeply nested values are truncated. score() requires a finite numeric value and accepts optional metadata (for example score("faithfulness", 0.4, metadata={"reason": "answer cites no context"})) that is redacted with the same rules before it is written to the score event. Generation token usage and generation cost require at least one field, and all provided values must be non-negative finite numbers.

Generation cost is user-provided. Bir records explicit cost values and defaults the currency to USD; it does not calculate provider pricing automatically.

Service Metadata

Use configure() to tag traces with the service and environment that produced them. Both values are optional, must be non-empty strings, and are recorded on trace root events under metadata.service so the server and dashboard can filter traces by service and environment.

from bir import configure

configure(service_name="rag-api", environment="production")

Sampling

Use configure(sample_rate=...) to keep local trace volume bounded under load. sample_rate is the probability (0.0 to 1.0) that a trace is recorded and defaults to 1.0, which records every trace. The decision is made once per trace root and inherited by every event under it, so a sampled-out trace and all of its spans, generations, tool calls, retrievals, and scores write nothing.

Sampling never changes control flow: a sampled-out function still runs and still raises its own exceptions; only the local JSONL writes are skipped.

from bir import configure

configure(sample_rate=0.1)  # record about 10% of traces

Retrieval

Use retrieval() to record RAG lookups with the existing tool_call event contract. It sets metadata.kind to retrieval, stores the query at input.query when input capture is enabled, and stores retrieved records at output.documents when output capture is enabled. Document rank values must be non-negative integers, and document score values must be non-negative finite numbers.

from bir import retrieval

with retrieval("vector_search", query=question) as result:
    result.add_document(
        id="doc-1",
        rank=1,
        score=0.82,
        source="docs",
        text="Bir records local traces with JSONL.",
    )

Prompt Versions

Use prompt() to attach prompt identity and version metadata to a generation. Prompt template text, variables, and rendered prompts are not captured unless you opt in.

from bir import generation, prompt

answer_prompt = prompt(
    "answer_question",
    version="v1",
    template="Answer using this context: {context}",
    variables={"context": "local context"},
)

with generation("local.llm", model="demo-model", prompt=answer_prompt) as gen:
    gen.set_output("ok")

The generation event records metadata.prompt.name, metadata.prompt.version, and a metadata.prompt.template_sha256 when a template is provided. To inspect the actual prompt payload locally, opt in explicitly:

answer_prompt = prompt(
    "answer_question",
    version="v1",
    template="Answer using this context: {context}",
    variables={"context": "local context"},
    capture_template=True,
    capture_variables=True,
    capture_rendered=True,
)

Captured prompt fields use the same best-effort redaction as other captured payloads. After sending traces to the local server, the dashboard shows prompt metadata on generation details without requiring you to inspect the raw event JSON.

LangChain Callback

Use BirCallbackHandler to record LangChain callback events as Bir traces without adding LangChain as a Bir dependency:

from bir import configure
from bir.integrations.langchain import BirCallbackHandler

configure(capture_inputs=True, capture_outputs=True)

result = chain.invoke(
    {"question": "What is Bir?"},
    config={"callbacks": [BirCallbackHandler()]},
)

Root chains become trace events, nested chains become spans, LLM/chat model callbacks become generation events, retrievers become retrieval tool calls, and tools become tool call events. Direct model calls without an active chain create a small implicit trace root. The handler records token usage from common LangChain response shapes including llm_output.token_usage, usage_metadata, and response_metadata.token_usage.

Mistral

Use trace_chat() to record Mistral chat completions without adding mistralai as a Bir dependency:

from bir import trace
from bir.integrations.mistral import trace_chat

with trace("chat"):
    response = trace_chat(
        client.chat.complete,
        model="mistral-small-latest",
        messages=[{"role": "user", "content": "What is Bir?"}],
    )

The wrapper forwards positional and keyword arguments unchanged, returns the Mistral response untouched, and records the response model, token usage, and model_dump() output when capture settings allow it.

Cohere

Use trace_chat() to record Cohere v2 chat calls without adding cohere as a Bir dependency:

from bir import trace
from bir.integrations.cohere import trace_chat

with trace("chat"):
    response = trace_chat(
        client.chat,
        model="command-a-03-2025",
        messages=[{"role": "user", "content": "What is Bir?"}],
    )

The wrapper forwards positional and keyword arguments unchanged, returns the Cohere response untouched, records the request model, and reads token usage from response.usage.tokens when present.

Local Evals And Experiments

Bir includes a small deterministic evaluation layer for local regression checks. It does not require a server or an LLM judge.

from bir.evals import Dataset, DatasetExample, contains, exact_match, latency_under, run_experiment


dataset = Dataset(
    [
        DatasetExample(
            id="q1",
            input={"question": "What is Bir?"},
            expected="An observability SDK",
        )
    ]
)


def answer_question(question: str) -> str:
    return "Bir is an observability SDK."


result = run_experiment(
    "quickstart",
    dataset=dataset,
    task=answer_question,
    evaluators=[
        contains(),
        exact_match("Bir is an observability SDK."),
        latency_under(1000),
    ],
)

print(result.aggregate_scores)

Datasets can be stored as JSONL:

{"id":"q1","input":{"question":"What is Bir?"},"expected":"An observability SDK"}

Load and run them locally:

from bir.evals import Dataset, contains, list_experiments, load_experiment, run_experiment, send_experiment

dataset = Dataset.from_jsonl("questions.jsonl")
result = run_experiment("prompt-v1", dataset=dataset, task=answer_question, evaluators=[contains()])
loaded = load_experiment(result.path)
summaries = list_experiments()

Dataset.to_jsonl() redacts common secret-like values by default when exporting examples. If you intentionally need to preserve raw dataset payloads, opt out explicitly:

dataset.to_jsonl("questions.jsonl", redact=False)

Experiment results are written to .bir/experiments/*.jsonl by default, with one result row per example. Bir also writes a sibling .summary.json file with the experiment id, status, example count, error count, aggregate scores, and result path so local runs can be listed without scanning every result row. Available deterministic evaluators are exact_match(), contains(), regex_match(), json_valid(), field_equals(), field_contains(), latency_under(), cost_under(), numeric_between(), retrieved_context_contains(), answer_context_overlap(), answer_contains_citation(), and custom_evaluator().

Upload a completed experiment to a running Bir server so the dashboard can show the experiment list and per-example result detail:

send_experiment(result.path, "http://127.0.0.1:8000")

To connect local experiment rows back to traces, opt in when running the experiment:

result = run_experiment(
    "prompt-v1",
    dataset=dataset,
    task=answer_question,
    evaluators=[contains()],
    record_traces=True,
)

record_traces=True writes one trace per dataset example and records evaluator outputs as score events on that trace.

Use threshold evaluators for local gates:

from bir.evals import cost_under, latency_under, numeric_between

evaluators = [
    latency_under(1000),
    cost_under(0.05),
    numeric_between(min_value=0.0, max_value=1.0),
]

latency_under() uses measured task duration from run_experiment(). cost_under() checks explicit cost fields returned by your task, either {"total_cost": 0.01} or {"cost": {"total_cost": 0.01}}. Bir records only the cost values you provide and does not calculate provider pricing. numeric_between() checks numeric task outputs, or a numeric field when field= is provided.

Use structured output evaluators for JSON-like task results:

from bir.evals import field_contains, field_equals, numeric_between

evaluators = [
    field_contains("answer", "observability"),
    field_equals("citations[0].id", "doc-1"),
    numeric_between(min_value=0.7, max_value=1.0, field="confidence"),
]

Field paths support dot paths and list indexes, such as answer, usage.total_tokens, and items[0].name. Missing paths produce a 0.0 score with failure metadata instead of failing the experiment.

Use custom_evaluator() for local checks that are specific to your task:

from bir.evals import EvalResult, custom_evaluator

has_citation = custom_evaluator(
    "has_citation",
    lambda output, expected: "[1]" in str(output),
)

debuggable = custom_evaluator(
    "debuggable",
    lambda output, expected: EvalResult(
        name="debuggable",
        value=1.0,
        metadata={"expected": expected},
    ),
)

Custom evaluators may return bool, int, float, or EvalResult. Exceptions from custom evaluator functions surface normally during development.

Use retrieved_context_contains() to check retrieval quality without an LLM judge:

from bir.evals import retrieved_context_contains

evaluators = [
    retrieved_context_contains("observability"),
]

retrieved_context_contains() reads the contexts list from a structured RAG output such as {"answer": "...", "contexts": ["doc text", ...]} and scores 1.0 when expected appears in one of the retrieved strings. Missing or empty contexts produce a 0.0 score with failure metadata instead of failing the experiment. Pass case_sensitive=False for case-insensitive matching. This is a deterministic retrieval check, not proof that the answer used the context.

Use answer_context_overlap() to flag answers that may not be grounded in the retrieved context, also without an LLM judge:

from bir.evals import answer_context_overlap

evaluators = [
    answer_context_overlap(0.5),
]

answer_context_overlap() reads the same structured RAG output ({"answer": "...", "contexts": ["doc text", ...]}) and scores 1.0 when at least min_ratio of the answer's word tokens also appear in the retrieved contexts. It is a deterministic faithfulness heuristic, not proof of faithfulness: paraphrased but faithful answers can score low, and unfaithful answers that reuse context words can score high. Missing answers or contexts produce a 0.0 score with failure metadata instead of failing the experiment.

Use answer_contains_citation() to check that an answer cites a source, also without an LLM judge:

from bir.evals import answer_contains_citation

evaluators = [
    answer_contains_citation(),
]

answer_contains_citation() reads a plain answer string or the answer field of a structured RAG output ({"answer": "...", "contexts": [...]}) and scores 1.0 when the answer contains a citation marker. By default any bracketed marker such as [1] or [doc-1] counts; pass pattern to require a custom citation format such as pattern=r"\(\d+\)" for markers like (1). This is a deterministic format check, not proof that the citation is correct or that the cited source supports the answer. Non-text output or a missing answer produces a 0.0 score with failure metadata instead of failing the experiment.

Event Loading

load_events() validates JSONL records against the current event schema and raises ValueError for malformed rows, unsupported event types, invalid timestamps, or unsupported schema versions.

To write traces somewhere else:

configure(trace_path="tmp/bir-traces.jsonl")

Development

Run the SDK unit tests from this directory:

PYTHONPATH=src ../../.venv/bin/python -m unittest discover -s tests

Or install the package with test dependencies and run pytest:

python3 -m pip install -e ".[dev]"
pytest

Run repository type checking from the repository root:

./.venv/bin/pyright

Run the release verification script from the repository root before publishing:

./.venv/bin/python scripts/verify_release.py

The script builds a temporary SDK wheel, installs it in a fresh temporary virtual environment, and smoke-tests local tracing and retrieval without writing build artifacts into the repository.

Release planning lives in CHANGELOG.md and docs/SDK_RELEASE_CHECKLIST.md.

License

Bir is source-available under the Functional Source License 1.1 with Apache 2.0 as the future license (FSL-1.1-ALv2). FSL is not an OSI-approved open source license.