pramana-ai 0.1.0

CLI tool for running LLM drift detection evaluations

pramana  ·  प्रमाण

Track whether LLM API outputs stay consistent over time.

Crowdsourced drift detection for LLM APIs. Run reproducible evals, compare results over time, catch silent model changes.

---

The Problem

When you call gpt-5 or claude-sonnet-4-6 today, you might get different behavior than yesterday. Providers update, fine-tune, and swap models behind stable identifiers. This is invisible.

There's no standard way to notice — let alone measure — these changes.

The Fix

git clone https://github.com/syd-ppt/pramana && cd pramana
uv pip install -e ".[dev]"

$ pramana run --tier cheap --model gpt-5.2
Running cheap suite against gpt-5.2...
✓ 10/10 passed
Pass rate: 100.0%

Same prompts. Same parameters. Deterministic where the provider allows it. Compare across runs and users.

---

Usage

# See all supported models
pramana models

# Run evals (auto-detects provider from model name)
export OPENAI_API_KEY=sk-...
pramana run --tier cheap --model gpt-4o

# Aliases work too
pramana run --tier cheap --model opus

# Submit to the community dashboard
pramana submit results.json

Tiers:

Tier	Tests	Purpose
cheap	10	Smoke test, CI gates
moderate	25	Regular monitoring
comprehensive	75	Full evaluation

All tiers cover 6 categories: reasoning, factual, instruction following, coding, safety, creative.

---

Providers

Provider	Temperature	Seed	Reproducibility
OpenAI	✅ Enforced	✅ Enforced	High
Anthropic	✅ Enforced	❌ Ignored	Low
Google	✅ Enforced	✅ Enforced	Medium

For scientific drift detection, use OpenAI API with explicit keys. See REPRODUCIBILITY.md.

---

How It Works

You run pramana ──► Fixed prompts hit the API ──► Results hashed & stored
                                                         │
Other users run pramana ──► Same prompts ──► Results compared
                                                         │
                                              Drift detected via
                                              output consistency tracking

• Content-addressable hashing — SHA-256 of (model, prompt, output) for deduplication
• Deterministic parameters — temperature=0.0, seed=42 enforced by default
• No normalization layer — raw API responses, not filtered through LiteLLM

---

Authentication (Optional)

pramana login          # GitHub/Google OAuth
pramana whoami         # Check status
pramana delete         # GDPR: delete all your data

No login required to run evals or submit results. Auth enables personalized tracking.

---

Development

git clone https://github.com/syd-ppt/pramana && cd pramana
uv pip install -e ".[dev]"
pytest tests/

Backend: pramana-api · Dashboard: pramana.pages.dev

---

Contributing

1. Add test cases — append to suites/v1.0/{tier}.jsonl
2. Add providers — subclass BaseProvider in src/pramana/providers/
3. Improve assertions — new types in assertions.py

See CONTRIBUTING.md.

---

pramana (प्रमाण) — Sanskrit for proof, evidence, valid knowledge

Docs · Dashboard · Issues