raidex 0.1.4

Self-contained CLI to measure an LLM against the Raidex Responsible-AI index

Raidex

Measure any LLM's Responsible-AI profile in your own environment: your own fine-tuned or self-hosted model, or a frontier model. raidex.ai

raidex scores a model across open Responsible-AI benchmarks (safety, fairness, factuality, security, machine ethics, robustness, privacy, and sycophancy) and reports a composite RAI Score plus per-dimension scores. The same measurement core also powers a public leaderboard of frontier models.

Quickstart

pip install raidex        # Python 3.10 to 3.13

Measure your own or self-hosted model against any OpenAI-compatible endpoint (vLLM, Ollama, TGI, and so on):

raidex eval --model http://localhost:8000/v1 --served-name my-model --tier A+B

Measure a frontier model with any litellm model string. The provider key is read from the matching env var, for example OPENAI_API_KEY:

export OPENAI_API_KEY=sk-...          # plus ANTHROPIC_API_KEY for the judges
raidex eval --model openai/gpt-5.2 --tier A+B

No account, no queue, no upload, and no dependency on Raidex servers. raidex prints per-dimension and composite RAI scores and writes a self-describing JSON that never leaves your machine.

What you get

=== Raidex RAI Score ===
  RAI Score : 63.5
  Coverage  : 9/9  🟣
  Dimensions:
    safety           71.2
    fairness_bias    35.3
    factuality       52.0
    ...

• Board-comparable. Identical in scale to the public leaderboard (same core, same benchmarks, same normalization), so you can place your model against the frontier.
• A self-describing result JSON. The model spec, per-benchmark pinned dataset versions, judge, sampling settings, and a timestamp, so a score is reproducible and traceable. It is written locally and nothing is uploaded.
• Honest coverage. The composite is the mean of the constituents you ran, reported as N/9. Benchmarks you skip, or that need a judge you did not configure, simply lower coverage; they never fake a number.

CLI reference

raidex eval --model ...                         # required: a litellm model id OR a local endpoint URL
            --served-name my-model              # the model name served by a local endpoint URL
            --tier A | B | A+B                  # A = 7 core benchmarks, B = +robustness/privacy (default A)
            --benchmarks bbq,strongreject       # explicit subset (overrides --tier)
            --limit 150                         # sample the big benchmarks (small ones always run full)
            --judge anthropic/claude-opus-4-8   # LLM judge for SimpleQA / XSTest / StrongREJECT
            --dry-run                           # print a cost estimate and exit
            --offline                           # use only cached data; never touch the network
            --output results.json               # where to write the result (default: <model>__.json)

raidex fetch-data                               # pre-download and cache all benchmark data (for offline / air-gapped use)

Judges. SimpleQA, XSTest, and StrongREJECT are graded by an LLM judge. Configure one with --judge (or RAIDEX_JUDGE_MODEL). If none is available, those three are skipped with a printed reason and honestly reduced coverage, not a failure.

Offline and air-gapped. Run raidex fetch-data on a networked machine to populate a local cache (pinned dataset versions), copy that cache across, and run with --offline for zero network access.

The benchmarks

raidex runs 9 benchmarks across 8 dimensions. The RAI Score is the mean of normalized constituent scores (0 to 100); coverage is reported as N/9.

Tier	Benchmark	Dimension	Pipeline
A	BBQ	Fairness & Bias	lm-eval (generative)
A	WMDP	Security	lm-eval (generative)
A	SimpleQA	Factuality	litellm + judge
A	StrongREJECT	Security (refusal)	litellm + rubric judge
A	ETHICS	Machine Ethics	lm-eval (generative)
A	XSTest	Safety (over-refusal)	litellm + judge
A	Sycophancy	Sycophancy	litellm (judge-free flip-rate)
B	AdvGLUE	Robustness	litellm (exact-match)
B	ConfAIde	Privacy	litellm (correlation)

See space/METHODOLOGY.md for the index design, generative-task creation, judging, sampling, normalization, and disclosures.

The public leaderboard

There is also a public board of frontier models, produced by the same core:

• Live leaderboard: https://huggingface.co/spaces/cloudronin/raidex-space
• Results dataset: https://huggingface.co/datasets/cloudronin/raidex-results
• Eval queue: https://huggingface.co/datasets/cloudronin/raidex-requests

Running the board, the Space, or reproducing the published numbers is a maintainer task. See docs/leaderboard.md.

Repository layout

• raidex/: the pip-installable raidex CLI (raidex/cli.py) over the pure raidex.core eval-and-score library. The core is the shared foundation; the CLI and the backend service are two thin frontends over it, which is why a local score matches the board.
• space/: the Hugging Face Space (Gradio leaderboard app). See docs/leaderboard.md.
• backend/: the eval service that produces the public board. See docs/leaderboard.md.

License

MIT. See LICENSE.