Rubric

Rubric points at your API, reads its OpenAPI schema, and hits it with realistic load — no manual test scripts to write. It uses an LLM to generate the actual request payloads (valid data, edge cases, and garbage inputs), fires them concurrently, and gives you latency percentiles plus AI notes on what broke and why.

How it works

1. Schema ingestion — pulls /openapi.json, sorts routes into buckets (does it need a body? does it depend on an ID from an earlier request?)
2. Payload synthesis — the LLM writes ~50 payloads per endpoint across three tiers, generated in parallel so this doesn't become the slow part
3. Load engine — async workers fire the requests and stream live numbers to your terminal as it runs
4. Report — latency percentiles, pass/fail per endpoint, and a short AI writeup of likely root causes for anything that failed

Route buckets

Payload tiers

Getting set up

cd rubric
pip install -r requirements.txt
# or install as a package so you get the `rubric` command:
pip install -e .

Then copy the env file and fill in your provider:

cp .env.example .env

LLM_PROVIDER=groq          # or openai

# Groq — fast, has a free tier, this is what most people use
GROQ_API_KEY=your_groq_key_here
GROQ_MODEL=meta-llama/llama-4-scout-17b-16e-instruct

# OpenAI, if you'd rather use that
OPENAI_API_KEY=your_openai_key_here
OPENAI_MODEL=gpt-4o
OPENAI_BASE_URL=https://api.openai.com/v1

Using it

The simplest case:

rubric run http://localhost:8000

Tuning the run:

rubric run http://localhost:8000 \
  --requests 500 \
  --concurrency 50 \
  --tier happy_path \
  --p95-threshold 200 \
  --error-rate 0.01

Run every tier in one go:

rubric run http://localhost:8000 --tier all --requests 1000

Just want to see what Rubric found without actually load testing:

rubric inspect http://localhost:8000

Skip the AI diagnostics step if you just want raw numbers fast:

rubric run http://localhost:8000 --no-diagnostics

Switch providers per run instead of editing .env:

rubric run http://localhost:8000 --provider groq
rubric run http://localhost:8000 --provider openai

Testing against a self-signed cert (local/dev)

Rubric checks the target's TLS certificate before sending login credentials, the same way a browser would — if it can't verify the cert, it won't send anything. That's fine for a real API, but it'll block you if you're testing against localhost or a dev server on a self-signed cert. In that case, opt in explicitly:

rubric run https://localhost:8443 --allow-self-signed

Only do this against something you control. It turns off the check that stops your credentials and tokens from being sent to an unverified server — don't use it against a real target over a network you don't fully trust.

Flags

What you get back

While it's running, you'll see something like this update live:

RUBRIC · 847 requests · 142.3 RPS · 5.9s
┌────────────────────────┬──────┬──────┬──────┬──────┬──────┬───────┐
│ Endpoint               │  RPS │  P95 │  2xx │  4xx │  5xx │ Total │
├────────────────────────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ POST /users            │  142 │  84ms│  891 │   23 │    6 │   920 │
│ GET  /users/{user_id}  │   89 │  31ms│  412 │    0 │    1 │   413 │
│ POST /products         │  201 │  19ms│ 1204 │   11 │    0 │  1215 │
└────────────────────────┴──────┴──────┴──────┴──────┴──────┴───────┘

And a summary once it's done:

┌────────────────────────┬────────────┬─────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┐
│ Endpoint               │ Status     │ RPS │  P50 │  P95 │  P99 │  2xx │  4xx │  5xx │ Err% │
├────────────────────────┼────────────┼─────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┤
│ POST /users            │ PASSED     │ 142 │ 34ms │ 84ms │201ms │  891 │   23 │    6 │0.67% │
│ GET  /users/{user_id}  │ DEGRADED   │  89 │ 12ms │312ms │ 67ms │  412 │    0 │    1 │0.24% │
│ POST /products         │ FAILED     │ 201 │ 18ms │ 44ms │ 98ms │  980 │   11 │   24 │2.10% │
└────────────────────────┴────────────┴─────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┘

SUITE FAILED
Passed: 1  Degraded: 1  Failed: 1
Total: 2507 requests   Duration: 17.6s   RPS: 142.4
Thresholds — P95 < 200ms, error rate < 1%

For anything that failed, you'll get a short explanation of the likely cause, not just a status code:

1. POST /users
   Payload: {"name": "", "email": "x@y.com", "role": ""}
   Cause: empty 'role' string hits a NOT NULL constraint at the DB layer.
   Fix: add a validator to reject empty strings for that field.

2. POST /products
   Payload: {"price": -1, "name": "Widget", "category": "tools"}
   Cause: negative price trips a CHECK constraint in Postgres.
   Fix: add Field(gt=0) to the price field in the schema.

Trying it against the sample API

There's a small FastAPI app included if you want to try Rubric without pointing it at a real project yet:

pip install fastapi uvicorn
uvicorn sample_api:app --reload

# then, in another terminal:
rubric run http://localhost:8000
rubric inspect http://localhost:8000

The JSON report

Every run writes rubric_report_<url>_<timestamp>.json to disk:

{
  "suite_result": "SUITE FAILED",
  "thresholds": { "p95_ms": 200, "error_rate_pct": 1.0 },
  "summary": {
    "total_requests": 2507,
    "duration_seconds": 17.6,
    "overall_rps": 142.4,
    "connection_drops": 0,
    "passed": 1,
    "degraded": 1,
    "failed": 1
  },
  "endpoints": [ ... ],
  "failure_diagnostics": [ ... ]
}

suite_result is meant to be read by CI — Rubric exits 1 on a failed suite.

Layout

rubric/
├── core/
│   ├── schema.py         # OpenAPI ingestion & route categorization
│   ├── synthesizer.py    # LLM payload synthesis
│   └── llm.py            # Groq / OpenAI client wrapper
├── engine/
│   └── runner.py         # Async load engine & live streaming
├── report/
│   └── reporter.py       # Reports & AI diagnostics
├── cli/
│   └── main.py           # Click entry point
├── sample_api.py         # Demo FastAPI server
├── requirements.txt
├── setup.py
└── .env.example