The Problem

When you call gpt-5 or claude-sonnet-4-6 today, you might get different behavior than yesterday. Providers update, fine-tune, and swap models behind stable identifiers. This is invisible.

There's no standard way to notice — let alone measure — these changes.

The Fix

git clone https://github.com/syd-ppt/pramana && cd pramana
uv pip install -e ".[dev]"

$ pramana run --tier cheap --model gpt-5.2
Running cheap suite against gpt-5.2...
✓ 10/10 passed
Pass rate: 100.0%

Same prompts. Same parameters. Deterministic where the provider allows it. Compare across runs and users.

Usage

# See all supported models
pramana models

# Run evals (auto-detects provider from model name)
export OPENAI_API_KEY=sk-...
pramana run --tier cheap --model gpt-4o

# Aliases work too
pramana run --tier cheap --model opus

# Submit to the community dashboard
pramana submit results.json

Tiers:

All tiers cover 6 categories: reasoning, factual, instruction following, coding, safety, creative.

Providers

For scientific drift detection, use OpenAI API with explicit keys. See REPRODUCIBILITY.md.

How It Works

You run pramana ──► Fixed prompts hit the API ──► Results hashed & stored
                                                         │
Other users run pramana ──► Same prompts ──► Results compared
                                                         │
                                              Drift detected via
                                              output consistency tracking

• Content-addressable hashing — SHA-256 of (model, prompt, output) for deduplication
• Deterministic parameters — temperature=0.0, seed=42 enforced by default
• No normalization layer — raw API responses, not filtered through LiteLLM

Authentication (Optional)

pramana login          # GitHub/Google OAuth
pramana whoami         # Check status
pramana delete         # GDPR: delete all your data

No login required to run evals or submit results. Auth enables personalized tracking.

Development

git clone https://github.com/syd-ppt/pramana && cd pramana
uv pip install -e ".[dev]"
pytest tests/

Backend: pramana-api · Dashboard: pramana.pages.dev

Contributing

1. Add test cases — append to suites/v1.0/{tier}.jsonl
2. Add providers — subclass BaseProvider in src/pramana/providers/
3. Improve assertions — new types in assertions.py

See CONTRIBUTING.md.