posthog-llm-doctor 0.1.0

Diagnose and repair why your LLM traces aren't showing up in PostHog.

PostHog LLM Doctor

Your LLM traces aren't showing up in PostHog? Find out why — in one command.

doctor check sends a controlled test span to your PostHog project, reads it back, and tells you whether your LLM telemetry is landing and, if not, exactly why — with a plain-English fix. No guessing, no digging through ingestion logs.

The bug it was built for: frameworks like OpenInference (Arize, LlamaIndex) send spans PostHog accepts (HTTP 2xx) but never classifies as generations — so your dashboard stays empty with no error. The Doctor names that (and nine other failure modes) with doctor check, and repairs it in flight with doctor fix (below).

---

Quickstart (under 5 minutes)

You need a PostHog project and its project API key (Project Settings → API key, starts with phc_).

# 1. Install the CLI (requires Python 3.11+) — the command is `doctor`
pipx install posthog-llm-doctor  # or:  uv tool install posthog-llm-doctor  /  pip install posthog-llm-doctor
# …bleeding edge from git:        uv tool install git+https://github.com/Ref34t/llm-doctor

# 2. Point it at your project (env var keeps the key out of your shell history)
export POSTHOG_PROJECT_KEY=phc_your_project_key

# 3. Diagnose
doctor check

That's it. You'll get a verdict like:

✗ FAIL — missing_gen_ai_operation_name

  endpoint   https://us.i.posthog.com/i/v0/ai/otel
  control    surfaced ✓
  repro      dropped ✗
  cause      The pipe works, but your framework's span shape is dropped: it has no
             gen_ai.operation.name, the attribute PostHog uses to classify a generation.
  remedy     Set gen_ai.operation.name (e.g. "chat") on your spans so PostHog recognises
             them as generations…

EU region? Add --endpoint https://eu.i.posthog.com/i/v0/ai/otel (or set $POSTHOG_OTEL_ENDPOINT).

Confirm ingestion, not just acceptance

A 2xx from PostHog only means accepted — not ingested (a bogus key still returns 2xx). To have the Doctor verify the span actually surfaced, give it read-back credentials too:

export POSTHOG_PERSONAL_API_KEY=phx_your_personal_key   # Personal API Keys, scope: query:read
export POSTHOG_PROJECT_ID=12345                          # numeric project id
doctor check

Without these, the Doctor still runs but reports ⚠ WARN — readback_not_configured (it won't claim "healthy" off a 2xx it can't confirm).

---

Repair mode — doctor fix

Diagnosis tells you why traces vanish. doctor fix repairs them in flight. It runs a local OTLP receiver; point your app's exporter at it, and every span flows on to PostHog — broken OpenInference spans get the missing gen_ai.operation.name (and their llm.* fields remapped to gen_ai.*) added on the way, correctly-shaped spans pass through untouched. Trace ids are preserved, so the repaired spans stay stitched into the right trace.

# 1. Start the Doctor (same project key as `doctor check`)
export POSTHOG_PROJECT_KEY=phc_your_project_key
doctor fix                      # listens on http://localhost:4318, forwards to PostHog

# 2. Point your app's OpenTelemetry exporter at the Doctor instead of PostHog
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
#    …then run your app as usual.

Each received span prints a one-line verdict so you can see the repair happen:

received openinference.repro → missing_gen_ai_operation_name → translated → forwarded (accepted)
received gen_ai.control       → healthy                      → forwarded unchanged (accepted)

EU region? doctor fix --endpoint https://eu.i.posthog.com/i/v0/ai/otel (or $POSTHOG_OTEL_ENDPOINT). Port in use? doctor fix --port 4319 (or $LLM_DOCTOR_PORT).

Fix mode reports accepted (2xx) per span; to prove a repaired span actually surfaced, run doctor check (with read-back creds) — or see the opt-in live test in tests/test_e2e.py.

---

What it checks

doctor check compares a control span (correctly shaped, should surface) against a repro span (shaped like a real framework, may not). The comparison isolates "your pipe is broken" from "your span shape is wrong". Every run ends in one of three states:

	Verdict	Means
✓	PASS	healthy — telemetry is reaching PostHog.
✗	FAIL	A confirmed problem: your spans are the wrong shape (missing_gen_ai_operation_name, the OpenInference case), PostHog rejected the send (ingest_rejected), or even correct spans aren't surfacing (control_not_surfaced).
⚠	WARN	Couldn't verify — no read-back credentials, a network/query read-back error, or an unexpected result. Not claimed as healthy or broken.

The exit code drops it into CI:

• 0 — healthy, or accepted-but-unverified (no read-back credentials configured)
• 2 — a read-back credential you must fix (readback_auth_failed)
• 1 — everything else: a confirmed problem (wrong shape / ingest rejected / not surfacing), or a run that couldn't be verified (network or query read-back error, unexpected result)

So doctor check && deploy proceeds on a clean or gracefully-degraded run, and stops on a confirmed failure or one it couldn't verify.

---

Why traces go missing

PostHog classifies a span as an LLM generation from one attribute: gen_ai.operation.name. OpenInference (Arize) spans carry openinference.span.kind + llm.* instead and omit gen_ai.operation.name, so they're accepted but silently dropped before classification.

• Root cause, with a runnable repro: PostHog/posthog#66614
• OpenInference semantic conventions: https://github.com/Arize-ai/openinference/blob/main/spec/semantic_conventions.md
• PostHog OTel ingestion: https://posthog.com/docs/ai-observability/installation/opentelemetry

Repairing these spans in flight (so you don't have to change your app) is the Doctor's next version — fix mode.

---

Configuration

Every value resolves in this order: CLI flag > environment variable > default.

Flag	Env var	Required	Default
--project-key	POSTHOG_PROJECT_KEY	yes — to send	—
--personal-api-key	POSTHOG_PERSONAL_API_KEY	for read-back	—
--project-id	POSTHOG_PROJECT_ID	for read-back	—
--endpoint	POSTHOG_OTEL_ENDPOINT	no	https://us.i.posthog.com/i/v0/ai/otel

Prefer env vars over flags so secrets stay out of your shell history. Run doctor check --help for the full list.

---

Development

git clone https://github.com/Ref34t/llm-doctor && cd llm-doctor
uv sync                 # install deps + dev tools
uv run pytest -q        # tests
uv run ruff check .     # lint
uv run doctor check     # run the CLI from source

---

Part of the PostHog LLM Doctor project — diagnose with doctor check, repair in flight with doctor fix.

Not affiliated with PostHog. This is an independent, community tool built for PostHog users. "PostHog" is a trademark of PostHog, Inc.