traceguard

Point-in-time correct LLM instrumentation — the time-integrity layer for LLM pipelines.

When you run LLMs over historical data — backtesting a signal, replaying a pipeline, re-scoring an archive — TraceGuard makes it structurally hard to accidentally use a model or prompt that did not yet exist at the point in time you are simulating.

It is not a dashboard or a gateway; it is the lower layer that guarantees the timeline underneath one. It interoperates with observability stacks (Langfuse / Phoenix via the optional traceguard[otel] exporter) rather than competing with them — see the OpenTelemetry → Langfuse/Phoenix guide and docs/POSITIONING.md.

• traceguard.registry.models — model registry with released_at / available_to_us_at; select_model(..., strict=...) with mandatory explicit mode (no default), so anachronistic choices fail loudly.
• traceguard.registry.prompts — git-tracked YAML prompt templates; load_prompt pins the content hash into every trace.
• traceguard.sdk.tracer — @tracer.trace decorator and tracer.span() context manager recording input hash, model/prompt versions, output, and perf into SQLAlchemy (SQLite by default).
• traceguard.sdk.normalizer — the single canonical normalize_input / input_hash (sorted keys, fixed float precision, normalized whitespace).
• traceguard.sdk.wrappers.anthropic — wrap_anthropic auto-instruments an Anthropic SDK client (extra: traceguard[anthropic]).
• traceguard.sdk.wrappers.openai — wrap_openai auto-instruments an OpenAI SDK client's chat.completions and responses calls (extra: traceguard[openai]).
• traceguard.registry.replay — curated, lockable replay sets (create_replay_set / add_replay_item / lock_replay_set / build_locked_replay_set); once locked, the store physically rejects any mutation (invariant 4).
• traceguard.validators.lookahead — invariant validators (validate_feature_as_of, validate_model_timing, validate_reference_timing, assert_replay_set_locked) that raise InvariantViolation; call them in pytest/CI to enforce all four look-ahead invariants.

All of the above are also re-exported from the top level (from traceguard import select_model, tracer, assert_replay_set_locked, ...); the deep paths stay valid as aliases. The package ships py.typed, so the annotations (including the Literal-typed select_model(..., strict=...)) reach type-checkers. Persistence is fail-open: a tracing/DB failure never breaks or masks the instrumented call (set TRACEGUARD_STRICT_PERSISTENCE=1 to fail closed).

Install

pip install traceguard

Requires Python 3.11+. Optional extras: pip install "traceguard[anthropic]" / pip install "traceguard[openai]" (Anthropic / OpenAI client wrappers) and pip install "traceguard[otel]" (OpenTelemetry / OpenInference export to Langfuse, Phoenix, or any OTLP backend).

Example

from datetime import datetime, timezone
from traceguard.registry.models import register_model, select_model
from traceguard.store.models import make_engine

engine = make_engine("sqlite:///traceguard.db")

register_model("demo-llm-2024", model_family="internal-ml",
               capability_class="general-llm",
               released_at=datetime(2024, 1, 10, tzinfo=timezone.utc),
               available_to_us_at=datetime(2024, 2, 1, tzinfo=timezone.utc),
               engine=engine)

# Backtesting as of mid-2025: models that arrived later are invisible.
model_id = select_model("general-llm",
                        available_at=datetime(2025, 6, 30, tzinfo=timezone.utc),
                        strict=True, engine=engine)

A complete runnable tour (synthetic data, no API keys) lives in examples/quickstart.

Contract

The binding interface contract — table schemas, SDK signatures, the four look-ahead invariants, SemVer rules — is in docs/SPEC.md.

Implemented: tracer, model/prompt registries, normalizer, all four look-ahead invariants (1–4, including locked replay sets), Anthropic + OpenAI wrappers, and a frozen public API. Not yet (post-1.0): drift checks + alerts, the full replay executor / A-B compare tooling, a CLI, Postgres/TimescaleDB, Voyage wrapper — see TRACEGUARD_ROADMAP.md.

Development

cd packages/traceguard
uv sync
uv run pytest        # 181 tests (4 skip without the contamination-hf extra)

License

Apache-2.0.