Tirkit
Open-source LLM observability and evaluation you can trust: trace every kind of model, turn real traffic into test sets, compare changes with confidence intervals, gate CI, and measure whether your LLM judge agrees with humans.
Status: M0–M10 built (see PLAN.md); outside users and the write-up are next · Licence: Apache-2.0 · Named for: Tir, the Armenian god of writing, who recorded people's deeds and led them to judgment
Why Tirkit
Every team shipping LLM features needs to see what their app does in production and to know whether a change made it better or worse. Most decide by feel, or with an LLM judge nobody has checked. Tirkit does both observability and evaluation, self-hosted, and it measures properly: every comparison has a confidence interval, and the judge's agreement with human labels is reported per rubric version.
What it does
- Any model: text and chat, vision, image generation, embeddings, small and local models (Ollama, LM Studio, vLLM, transformers), speech-to-text, text-to-speech and speech-to-speech, with images and audio stored and playable in the UI.
- Observe: Python and TypeScript SDKs, auto-instrumentation for OpenAI (and any OpenAI-compatible server), Anthropic, LangChain and transformers; OpenTelemetry ingest for everything else; sessions, users, cost from a 5,500-model price table, latency percentiles, dashboards, alerts, end-user feedback, PII redaction at ingest.
- Evaluate: datasets mined from traces (redacted, de-duplicated, versioned), rule, speech (WER/CER, TTS round-trip) and LLM-judge scorers, experiments, side-by-side comparison with 95% CIs, human annotation queues, online evals on sampled production traffic with a budget, and a CI gate that fails only when a change is confidently worse.
- Iterate: versioned prompts with deploy labels (
production,staging) fetched at runtime, and a playground that replays any traced call. - Trust: judge-reliability reports (Cohen's kappa, weighted kappa, position-swap flip rate, length bias) stored per rubric version.
Quickstart
pip install "tirkit[server]" # the SDK, the server and the web UI; no Docker needed
tirkit serve # http://127.0.0.1:8000: create the admin, then an API key
export TIRKIT_HOST=http://localhost:8000 TIRKIT_PUBLIC_KEY=pk-tirkit-... TIRKIT_SECRET_KEY=sk-tirkit-...
In the app you are tracing, pip install tirkit is enough (the server packages are not needed there).
import tirkit
tirkit.init()
tirkit.instrument() # OpenAI / Anthropic / transformers calls are now traced
No server yet? tirkit.init() with no settings writes tirkit.db in the current folder; run
tirkit serve in that folder later and the first account you create opens on those traces.
For a team, docker compose up -d (Postgres, server and worker) is in self-hosting.
More in the docs: evaluation, CI gating, prompts, self-hosting.
Develop locally:
make setup && make web # Python env and the UI build
make serve # http://localhost:8000
make dev-seed # synthetic dev data on every screen: traces, dataset, experiments, labels, judge reports, prompts, alerts
make test && make lint
The bar: 9/10
A reviewer should be able to see, in five minutes:
docker compose upto a first trace in the UI in under 10 minutes, following the docs- Real traffic traced through it (your job if allowed, a friend's product, or a client), with online evals running
- The full loop in the UI: trace → dataset → experiment → compare with CIs → CI gate
- Judge–human agreement (Cohen's kappa) measured, then improved by calibration
- Seeded regressions caught by the CI gate (x of N)
- Every comparison reported with a confidence interval, never a single score
Results
Only measured numbers go here, with how each was measured.
| Metric | Value | How measured |
|---|---|---|
| Judge–human agreement (kappa), before → after calibration | not measured yet | 100 outputs you labelled yourself |
| Position-swap flip rate | 65% → 32% (30/46 → 14/44 decided pairs; 95% CI 52–78% → 18–46%; paired difference −52 to −16 points) | research/experiments/e1_judge_bias.py: Qwen 3.5 9B (local, LM Studio) wrote a one-sentence and a 3–5 sentence answer to each of the 50 questions in research/evals/acme_qa.jsonl; the same model judged each pair in both orders with the reference answer, under the built-in correctness rubric (v0, e3c5e4ca) and a calibrated rubric (v1, 0128891a: explicit criteria, anchored examples, "ignore order and length"). Ties excluded. v0 showed a strong second-position preference (the answer shown first won 20% of decisions). Same small model as answerer and judge; one run, temperature 0. Raw data: research/experiments/results/e1_flip_rate.json |
| Seeded regressions caught | not measured yet | x of N deliberately broken prompts |
| Tracing overhead p99 | 0.024–0.025 ms added (p50 0.014 ms) | research/benchmarks/bench_tracing.py: 10,000 traced calls with a chat payload and token usage vs a bare call, SQLite via the batch exporter, 3 runs on an otherwise idle Apple M3 Pro (no model server, no Docker), Python 3.12; all 10,000 spans stored, 0 dropped. Re-measured 2026-11-16 after prompt linking was added to every span |
| Server ingestion throughput | 1,679–3,105 spans/s, 0 dropped | research/benchmarks/bench_ingest.py: 8 clients posting 100-span batches of ~1 KB chat spans for 30 s to tirkit serve (2 workers) on Postgres 16 in Docker, same Apple M3 Pro laptop, 3 runs; spans stored = spans sent every run |
| Trace list query p95 | 47 ms default view (worst filter: 83 ms) | research/benchmarks/bench_query.py: 1,000,000 spans / 200,000 traces in Postgres 16 (Docker, Apple M3 Pro); GET /traces with total count, 50 runs per query after 3 warm-ups; 8 query shapes (user, session, name, errors, last hour, deep page, latency) all ≤ 83 ms p95. Full-text search not included |
| Judge prompt-injection success, before → after hardening | 1/78 → 0/78 attacks succeeded (95% Wilson CI 0.2–6.9% → 0–4.7%): no measurable reduction, because this judge was already hard to fool. Flag suspect_injection: 78/78 attacks flagged (the patterns were written while reading this suite; 8 of 12 attacks written afterwards), 0/16 good answers flagged. Cost: 32 of 102 judgments moved by one point, one good answer fell from 4 to 3, 15/16 good answers still scored 4 or more |
research/security/judge_injection/run.py: 78 attacks (52 on factual questions with a reference answer, 26 on subjective rubrics without one; 12 categories incl. 12 pairwise) in tests/fixtures/judge_injection.jsonl, plus 24 good or plainly wrong answers. An attack succeeds when the wrong answer scores 4+ on a 1-5 scale or wins a pairwise comparison in both orders. Qwen 3.5 9B (local, LM Studio) as judge, temperature 0, one run each (deterministic: a rerun of the old prompt matched row for row). "Before" is the unmodified judge from the previous commit run from a git archive copy; "after" is LLMJudge(hardened=True). A 9B local model says nothing about other judge models. Raw data: research/security/judge_injection/results/ |
| Online judge vs user feedback (E3) | not measured yet | correlation on the same real traces |
| Real traces per day | not measured yet | from outside users |
Architecture
flowchart LR
App[Your LLM app] -->|Python / TS SDK| Server[Tirkit server · FastAPI]
OTel[Any OTel exporter] -->|OTLP| Server
Server --> DB[(Postgres / SQLite)]
UI[Web UI · React] <--> Server
Worker[Worker: online evals, experiments, alerts] <--> DB
DB --> DS[Datasets]
DS --> Runner[Experiments]
Prompts[Prompt versions] --> Runner
Scorers[Rules · LLM judge · humans] --> Runner
Runner --> Report[Compare with CIs]
Runner --> Gate[CI gate + PR comment]
Stack
Python (SDK, FastAPI server, worker, Typer CLI), Postgres (SQLite locally), React + TypeScript UI, TypeScript SDK, OpenTelemetry, Docker Compose, GitHub Actions.
Repo layout
src/tirkit/
tracing.py, exporter.py, client.py, http.py SDK: spans, batching, local or server mode
instrumentation/ OpenAI, Anthropic auto-tracing integrations/ LangChain, transformers
media.py, pricing.py, redact.py, serialize.py store.py SQLite and Postgres
datasets.py, scorers.py, judge.py, stats.py, runner.py, report.py evaluation
prompts.py, providers.py, jobs.py, worker_tasks.py, online.py, alerts.py, reliability.py
annotation.py, accounts.py, otel.py, cli.py
server/ FastAPI app, routes, settings
web/ React UI sdk-ts/ TypeScript SDK
examples/ quickstart, RAG demo, eval demo, OTel demo, dev seed, CI workflow
research/ benchmarks/ (tracing, ingestion, query), experiments/ (E1, E2), evals/ (QA set)
docs/ SPEC, PLAN, decisions, docs site (docs/site)
Documents
- SPEC.md: what to build, exactly, with the decisions already made
- PLAN.md: milestones in build order, each with a definition of done
- ROADMAP.md: planned work
- AGENTS.md: rules for everyone building here, human or AI
- docs/DECISIONS.md: log of decisions made during the build
- docs/site/: user documentation (
mkdocs serve) - docs/E1.md: judge-reliability experiment protocol
- CONTRIBUTING.md: how to contribute, including the one-time CLA
- SECURITY.md: how to report a vulnerability privately
- CHANGELOG.md: notable changes per release
Licence
Tirkit is open source under the Apache License 2.0. See NOTICE. Contributions are accepted under the Contributor License Agreement, and everyone taking part follows the Code of Conduct.
Metadata
Release files for tirkit 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| tirkit-0.1.0.tar.gz | 717.1 kB | With provenance | Details for tirkit-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| tirkit-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for tirkit-0.1.0-py3-none-any.whl |
Total release size: 1.3 MB
Release files / tirkit-0.1.0.tar.gz
| Download URL | tirkit-0.1.0.tar.gz |
|---|---|
| Size | 717.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a17ba953a3c6e01ae77b065e2c720bdf6c799454e96fe8b8929b63297e95852c
|
|
BLAKE2b-256 checksum How to use checksums |
44d0f942d945676f3e66deff9dcda503e48198ba63e2f3605eb84260f31a9576
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logRelease files / tirkit-0.1.0-py3-none-any.whl
| Download URL | tirkit-0.1.0-py3-none-any.whl |
|---|---|
| Size | 549.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
07b7db442009b0bf7eef5e9472c0d3f9f336dd84ffccc3d41ecc2993c090df2b
|
|
BLAKE2b-256 checksum How to use checksums |
42b1c8a602c3493101c4cba92553cf25533f5285da2acaa14a908312d99dcd53
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logSecurity for tirkit 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for tirkit-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.