gaunt 0.1.0

Python SDK for logging LLM traces to an analytics scoring API.

LLM Analytics Python SDK

Python SDK for collecting LLM app traces (chat, RAG, vision, structured output), standardizing them, and sending them to an Analytics Bank API for scoring.

Features

• Pydantic v2 strict payload models.
• Sync and async clients via httpx.
• Built-in scoring tags via ScoringTag.
• Automatic UUID generation for trace_id.

Installation

pip install gaunt

For local development:

pip install -e ".[dev]"

Quickstart

from gaunt import Client, ScoringTag

client = Client(
    api_key="sk_live_xxx",
    base_url="https://analytics-bank.example.com",
)

result = client.log(
    project_id="project_123",
    inputs={
        "messages": [
            {"role": "system", "content": "You are a concise assistant."},
            {"role": "user", "content": "What is RAG?"},
        ],
        "rag_context": [
            {"content": "RAG combines retrieval and generation.", "source_id": "doc-1"},
        ],
    },
    outputs={
        "raw": "RAG uses retrieved knowledge to ground generations.",
    },
    expected={"topic": "rag"},
    metadata={"env": "prod"},
    scoring_tags=[
        ScoringTag.RAG_FAITHFULNESS,
        ScoringTag.ANSWER_CORRECTNESS,
    ],
)

print(result)
client.close()

Scoring Tags

• ScoringTag.SCHEMA_VALIDITY -> score:schema_validity
• ScoringTag.RAG_FAITHFULNESS -> score:rag_faithfulness
• ScoringTag.VISION_HALLUCINATION -> score:vision_hallucination
• ScoringTag.TOXICITY -> score:toxicity
• ScoringTag.NUMERIC_SCORING -> score:numeric_scoring
• ScoringTag.ANSWER_CORRECTNESS -> score:answer_correctness

Payload Shape

The SDK emits payloads in this form:

{
  "project_id": "project_123",
  "trace_id": "uuid...",
  "inputs": {
    "messages": [],
    "images": [],
    "rag_context": [],
    "json_schema": {}
  },
  "outputs": {
    "raw": "...",
    "parsed": {}
  },
  "expected": {},
  "metadata": {},
  "config": {
    "scoring_tags": ["score:rag_faithfulness", "score:schema_validity"]
  }
}

Testing

pytest -q

Publishing

See docs/publishing.md for a release checklist and TestPyPI/PyPI upload commands.