Rubric

Rubric is a CLI tool for testing APIs using their OpenAPI schema.

It reads the API's /openapi.json, generates request payloads with an LLM, sends the requests concurrently, and records response times and status codes. It can also use the failed requests to generate a short diagnostic report.

How it works

1. Read the OpenAPI schema Rubric loads /openapi.json and identifies the available routes, methods, parameters, and request bodies.

2. Generate test payloads An LLM generates payloads for each endpoint. The payloads are split into valid inputs, edge cases, and malformed inputs.

3. Run the requests Requests are sent asynchronously with a configurable concurrency limit. Progress and response statistics are shown in the terminal.

4. Generate the report Rubric records latency percentiles, response status codes, error rates, and endpoint results. Failed requests can also be passed to the LLM for a short explanation.

Route buckets

Bucket	Methods	Description
A	GET, DELETE	Endpoints without a request body
B	POST, PUT, PATCH	Endpoints that accept a request body
C	Any	Endpoints that need an ID created by an earlier request

Payload tiers

Tier	Count	Description
happy_path	17	Normal values that should be accepted
edge_cases	17	Boundary values, empty strings, long strings, large numbers, Unicode, etc.
malformed	16	Wrong types, missing fields, and invalid structures

Installation

Clone the repository and install the dependencies:

cd rubric
pip install -r requirements.txt

To install Rubric as a CLI command:

pip install -e .

Copy the environment file:

cp .env.example .env

Then configure the LLM provider:

LLM_PROVIDER=groq

GROQ_API_KEY=your_groq_key_here
GROQ_MODEL=meta-llama/llama-4-scout-17b-16e-instruct

OPENAI_API_KEY=your_openai_key_here
OPENAI_MODEL=gpt-4o
OPENAI_BASE_URL=https://api.openai.com/v1

Usage

Run against a local API:

rubric run http://localhost:8000

Specify the request count, concurrency, payload tier, and thresholds:

rubric run http://localhost:8000 \
  --requests 500 \
  --concurrency 50 \
  --tier happy_path \
  --p95-threshold 200 \
  --error-rate 0.01

Run all payload tiers:

rubric run http://localhost:8000 --tier all --requests 1000

Inspect the API without running the load test:

rubric inspect http://localhost:8000

Disable the LLM diagnostic step:

rubric run http://localhost:8000 --no-diagnostics

Choose a provider for a single run:

rubric run http://localhost:8000 --provider groq
rubric run http://localhost:8000 --provider openai

Self-signed certificates

For local development servers using a self-signed certificate:

rubric run https://localhost:8443 --allow-self-signed

This should only be used against a server you control. The option allows Rubric to continue when the target certificate cannot be verified.

Command options

Flag	Default	Description
--requests	100	Total number of requests
--concurrency	20	Maximum number of concurrent requests
--tier	happy_path	happy_path, edge_cases, malformed, or all
--p95-threshold	200	P95 latency threshold in milliseconds
--error-rate	0.01	Maximum allowed 5xx error rate
--no-diagnostics	False	Disable LLM failure diagnostics
--save	True	Save the JSON report
--provider	.env value	Override the configured LLM provider
--allow-self-signed	False	Allow unverified TLS certificates

Output

During a run, Rubric displays the current request and response statistics:

RUBRIC · 847 requests · 142.3 RPS · 5.9s

┌────────────────────────┬──────┬──────┬──────┬──────┬──────┬───────┐
│ Endpoint               │ RPS  │ P95  │ 2xx  │ 4xx  │ 5xx  │ Total │
├────────────────────────┼──────┼──────┼──────┼──────┼──────┼───────┤
│ POST /users            │ 142  │ 84ms │ 891  │ 23   │ 6    │ 920   │
│ GET /users/{user_id}   │ 89   │ 31ms │ 412  │ 0    │ 1    │ 413   │
│ POST /products         │ 201  │ 19ms │ 1204 │ 11   │ 0    │ 1215  │
└────────────────────────┴──────┴──────┴──────┴──────┴──────┴───────┘

At the end of the run, each endpoint is classified as PASSED, DEGRADED, or FAILED based on the configured thresholds.

┌────────────────────────┬────────────┬─────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┐
│ Endpoint               │ Status     │ RPS │ P50  │ P95  │ P99  │ 2xx  │ 4xx  │ 5xx  │ Err% │
├────────────────────────┼────────────┼─────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┤
│ POST /users            │ PASSED     │ 142 │ 34ms │ 84ms │201ms │ 891  │ 23   │ 6    │0.67% │
│ GET /users/{user_id}   │ DEGRADED  │ 89  │ 12ms │312ms │ 67ms │ 412  │ 0    │ 1    │0.24% │
│ POST /products         │ FAILED     │ 201 │ 18ms │ 44ms │ 98ms │ 980  │ 11   │ 24   │2.10% │
└────────────────────────┴────────────┴─────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┘

SUITE FAILED

Passed: 1
Degraded: 1
Failed: 1

Total: 2507 requests
Duration: 17.6s
RPS: 142.4

Thresholds:
P95 < 200ms
Error rate < 1%

With diagnostics enabled, failed requests can also include an explanation:

1. POST /users

   Payload: {"name": "", "email": "x@y.com", "role": ""}

   Cause: empty 'role' string reaches the database layer.

   Fix: validate the field before sending the request.

Sample API

The repository includes a small FastAPI application for testing Rubric locally.

Install FastAPI and Uvicorn:

pip install fastapi uvicorn

Start the sample API:

uvicorn sample_api:app --reload

Then run Rubric in another terminal:

rubric run http://localhost:8000

Or inspect the API first:

rubric inspect http://localhost:8000

JSON reports

Rubric saves a JSON report after each run:

rubric_report_<url>_<timestamp>.json

Example:

{
  "suite_result": "SUITE FAILED",
  "thresholds": {
    "p95_ms": 200,
    "error_rate_pct": 1.0
  },
  "summary": {
    "total_requests": 2507,
    "duration_seconds": 17.6,
    "overall_rps": 142.4,
    "connection_drops": 0,
    "passed": 1,
    "degraded": 1,
    "failed": 1
  },
  "endpoints": [],
  "failure_diagnostics": []
}

SUITE FAILED is also used as the CI result. Rubric exits with status code 1 when the configured thresholds are exceeded.

Project structure

rubric/
├── core/
│   ├── schema.py
│   ├── synthesizer.py
│   └── llm.py
├── engine/
│   └── runner.py
├── report/
│   └── reporter.py
├── cli/
│   └── main.py
├── sample_api.py
├── requirements.txt
├── setup.py
└── .env.example

Main components

• core/schema.py - reads the OpenAPI schema and categorizes routes
• core/synthesizer.py - generates request payloads
• core/llm.py - LLM provider handling
• engine/runner.py - sends requests and collects metrics
• report/reporter.py - builds reports and diagnostics
• cli/main.py - CLI entry point
• sample_api.py - local API used for testing

Security notes

Data sent to the LLM

When diagnostics are enabled, Rubric sends information about failed requests to the configured LLM provider.

This includes:

• the endpoint
• a redacted request payload
• the first 300 characters of the response

Rubric attempts to remove values from fields such as password, token, and similar secret-looking fields. It also looks for JWTs, bearer tokens, API keys, and email addresses.

This redaction is not guaranteed to catch every sensitive value. If the target API can return sensitive information in error responses, use:

rubric run http://localhost:8000 --no-diagnostics

Target reset

Rubric does not call /reset by default.

The reset endpoint is included for the sample API. If you need it:

rubric run http://localhost:8000 --reset-target

or:

RUBRIC_ALLOW_RESET=1

Only enable this when testing an environment where resetting the data is acceptable.

Credentials

Rubric stores its local configuration and authentication files with owner-only permissions:

~/.rubric/config.json
.rubric/auth.json

For CI or other non-interactive runs, stored passwords are removed after the run.

For authentication, prefer:

RUBRIC_AUTH_TOKEN=...

over passing a token directly on the command line.

Reports

Values originating from the target API or the LLM are HTML-escaped before being included in reports.

The generated report also includes a restrictive Content Security Policy.