RedCrown

Localized model benchmarking with receipts. Run head-to-head evals across every model, provider, and config on your own data, locally, then turn the results into ranked, receipted proof you can hand to a client, a CFO, or a regulator.

Agents and harnesses run benchmarks for free. RedCrown is the neutral layer that turns them into a decision you can defend, and keeps re-proving it as prices and models change. Run anywhere, prove here.

Install

pip install redcrown

Requires Python 3.11+. The CLI has a small dependency footprint; server extras (pip install "redcrown[server]") are only needed if you run the API yourself.

Quickstart (no keys needed)

Try the bundled offline demo in about five seconds, no provider keys required:

redcrown eval --sample extract

This runs a small bundled extraction sample using stub outputs. The output says demo mode because no real models are called. Connect your provider keys in ~/.redcrown/credentials.json (or via redcrown login) for real quality scores.

Bring your own data (text tasks)

If you have a CSV or JSONL file of inputs, import it directly:

redcrown build-dataset --from-csv mydata.csv --task classify --out exp.json
redcrown eval exp.json --report-json out.json

CSV/JSONL columns:

--task values: extract, classify, summarize, qa

The importer fills in the provider scaffold for you. Edit exp.json to swap in the candidates you want to compare before running.

The exp.json schema

Every eval is a plain JSON file. Here is a minimal 2-candidate text example you can copy, edit, and run:

{
  "name": "classify-support-tickets",
  "quality_metric": "similarity",
  "quality_bar": 0.75,
  "reference_source": "labels",
  "pipeline": [
    {
      "id": "step1",
      "order": 1,
      "name": "classify",
      "input_type": "text",
      "output_type": "text",
      "input_binding": "dataset",
      "incumbent_candidate_id": "gpt4o-mini",
      "candidates": [
        {
          "id": "gpt4o-mini",
          "provider": "openai",
          "model": "gpt-4o-mini",
          "label": "GPT-4o Mini (incumbent)"
        },
        {
          "id": "llama-8b",
          "provider": "groq",
          "model": "llama-3.1-8b-instant",
          "label": "Llama 3.1 8B via Groq"
        }
      ]
    }
  ],
  "dataset": [
    {
      "id": "row1",
      "payload": { "kind": "text", "text": "My invoice is wrong." },
      "reference": "billing"
    },
    {
      "id": "row2",
      "payload": { "kind": "text", "text": "App crashes on login." },
      "reference": "bug"
    }
  ]
}

Required fields: name, quality_metric, quality_bar, reference_source, pipeline[].incumbent_candidate_id, pipeline[].candidates[].id, pipeline[].candidates[].provider, pipeline[].candidates[].model, dataset[].id, dataset[].payload.text.

No labels? Use the incumbent as reference

If you do not have labeled ground truth, leave reference off your dataset rows and set reference_source: "incumbent":

{
  "reference_source": "incumbent",
  ...
  "dataset": [
    { "id": "row1", "payload": { "kind": "text", "text": "My invoice is wrong." } }
  ]
}

RedCrown scores every candidate against your current model's own output, so you only need to beat or match what you run today. This is the fastest path to a cost proof when you have real traffic but no labeled examples.

Running a full eval

# run the fan-out locally, on your machine, with your keys
redcrown eval exp.json --report-json out.json

# (optional) sign in once per machine via device-code OAuth
redcrown login

# (optional) push the results to a shareable, no-login proof page
redcrown push out.json --proof-link

redcrown eval ranks every config on cost, quality, and latency against your own ground truth and names the cheapest one that clears your quality bar. Example:

RANKED  transcription · cheapest config at or above your 0.85 quality bar
  deepgram · nova-3-medical    quality 0.883    $294/mo   winner, 40% cheaper
  aws · transcribe-standard    quality 0.879    $487/mo   incumbent
  openai · whisper-1           quality 0.820    $122/mo   below your bar

That run is published as a live, no-login proof page: https://app.redcrown.ai/proof/O9iYVdWuaYjaL6mnImeIsD6TB1W_S6h4Frbx04YqAYQ

Note on build-dataset primock57: this command downloads the PriMock57 clinical-transcription corpus, which requires git-lfs. It is an audio eval designed for transcription benchmarking, not a general first step. Start with redcrown eval --sample extract or build-dataset --from-csv instead.

Free by construction

Evals run on your machine with your own provider keys, so RedCrown never sees your raw data and the run costs you nothing beyond your own inference. Only the results you choose to push become a cloud proof. --no-receipts keeps raw outputs local and uploads aggregates only.

Already ran an eval elsewhere?

You do not have to run anything through RedCrown to get a proof. Take the results from an eval you already ran, as a JSON in the RedCrown results format, and push them:

redcrown push results.json --proof-link

You get the same ranked, receipted, shareable report. The fastest path, with no install, is the web app at https://app.redcrown.ai/upload.

Eval your own OpenAI-compatible endpoint locally

If you have a private or self-hosted API that speaks the OpenAI chat-completion format, point redcrown eval at it without ever uploading your endpoint secret:

export REDCROWN_CUSTOM_HTTP='{"base_url":"https://api.example.com/v1","auth_header_name":"Authorization","auth_value":"Bearer sk-...","body_style":"openai"}'
redcrown eval exp.json

Set a candidate in your exp.json with "provider": "custom_http" and any "model" string your endpoint recognises. The JSON connection stays local; it is read directly from the env var at eval time and is never persisted or uploaded.

For coding agents (MCP)

Coding agents (Claude, Cursor, Codex) drive the whole loop over the hosted MCP server at mcp.redcrown.ai: scaffold an experiment, run it, review outputs, and mint a proof. The server is open source: https://github.com/RedCrown-ai/redcrown-mcp

Links

• Site: https://redcrown.ai
• App: https://app.redcrown.ai
• MCP server (open source): https://github.com/RedCrown-ai/redcrown-mcp

License

Proprietary. (c) RedCrown.ai