Lanternfish Python SDK

Record what your AI agent does: its LLM calls, its own steps, the decisions it makes and the release it runs on. Lanternfish compares those runs over time and tells you what changed.

Built on OpenTelemetry. Works next to an existing OpenTelemetry or Logfire setup.

Install

pip install "lanternfish[openai]"

Use the extra for each provider you call: openai, anthropic, gemini, groq, langchain, crewai, llamaindex, or all. Requires Python 3.11+.

Start

export LANTERNFISH_API_KEY="lfsk_..."            # Settings → API keys
export LANTERNFISH_ENDPOINT="https://your-lanternfish-endpoint"

import lanternfish

lanternfish.instrument(agent="travel-agent")

From here on, calls to installed providers are recorded. The SDK never picks a destination by itself: without an endpoint it stops with an error that says what to set.

Record your own steps and decisions

import lanternfish

@lanternfish.trace                                  # records arguments; capture_args=False for private data
def search_flights(destination: str) -> list[str]:
    ...

with lanternfish.set_user("user_42", session_id="sess_9"):   # removed again when the block ends
    with lanternfish.span("book-trip"):
        options = search_flights("Lisbon")
        with lanternfish.reasoning(name="pick_flight") as r:
            r.consider_option("KLM 1724", score=0.8, reason="Arrives before the meeting")
            r.decide("KLM 1724", confidence=0.8, reason="Arrives before the meeting")

A step that raises is marked failed, with the exception type and message.

Releases

Every span carries the release it ran on: release= if you pass it, otherwise LANTERNFISH_RELEASE, otherwise the commit your CI or host exposes (GITHUB_SHA, VERCEL_GIT_COMMIT_SHA, RENDER_GIT_COMMIT, …). To mark the moment a release went live, call this from your deploy step:

lanternfish.record_deployment("v1.8.2", metadata={"commit": "a1b2c3d", "environment": "production"})

Release verdict: stop a deploy that regressed

Once a release and the one before it each have 30 runs, Lanternfish compares them: run duration, success rate and cost per run, plus how the agent uses each tool (how many runs call it, how many times per run). The verdict is one of regressed, changed, improved, no_change or insufficient_data, and every finding states both sides:

v1.8.2 regressed against v1.8.1.
  - Success rate is down 18%: 80.0% vs 98.0% in v1.8.1.
  - search_flights is called 2.3 times per run, 1.1 in v1.8.1.
Based on 212 runs of v1.8.2 vs 1,840 runs of v1.8.1.

Gate a deploy on it from CI, after the canary has served traffic. The command exits 1 when the release regressed:

lanternfish verdict --release v1.8.2                       # gate on quality regressions
lanternfish verdict --release v1.8.2 --fail-on changed     # also on behavior changes
lanternfish verdict --release v1.8.2 --require-data        # also when there are not enough runs yet

Exit codes: 0 passed, 1 gate closed, 2 no verdict (unknown release, rejected key, API unreachable). From Python: lanternfish.get_verdict("v1.8.2").regressed. The verdict endpoint accepts the project API key and returns only aggregate numbers and tool names, never run content. The dashboard shows the same verdict under Releases.

Options

What is recorded

• LLM calls: model, tokens, duration, errors and, unless capture_content=False, the prompt and completion text.
• HTTP (only with include=["http"]): method, host, path and status. Never bodies, headers or query strings.
• SQL (only with include=["sql"]): the statement with literals replaced by ?. Never parameter values.

Short scripts and serverless

Spans are sent in the background in batches. Call lanternfish.flush() before a short script or a serverless handler returns; lanternfish.shutdown() runs on process exit by itself.

Troubleshooting

import logging
logging.basicConfig(level="INFO")

The lanternfish logger names the endpoint, lists the integrations it turned on, and logs every rejected batch with the reason (invalid key, plan limit reached, endpoint unreachable). Sending never raises into your code; a temporary error is retried once.

License

MIT