RedCrown

RedCrown helps teams check AI changes against defined requirements and inspect the evidence behind the result. Use your own cases and requirements to compare a changed AI feature with a reference run or supported accepted baseline. Inspect failures, understand missing evidence, and share the available results with the people making the decision.

The check runs on your machine. redcrown score and redcrown check read files, make no network call, and need no account and no provider key.

Install

pip install redcrown

The CLI installs with a small dependency footprint. The AWS Transcribe and Bedrock providers need pip install "redcrown[transcribe]"; the API needs pip install "redcrown[server]". Requires Python 3.11+.

Quickstart: check a change against a reference run

The example is synthetic: 40 invented support tickets, one label each, and the outputs of an imaginary ticket router at three revisions. No model ran. Every number below comes from the released CLI.

base=https://redcrown.ai/docs/quickstart/regression-check
curl -sSO $base/score-request.json -O $base/other-feature.json -O $base/policy.json -O $base/changed_cases.py

Score every revision's outputs against the labels. The evidence file is private: it holds the inputs, outputs and references.

$ redcrown score score-request.json --evidence-json evidence.json
Experiment ticket-router  objective=quality  bar=0.95  baseline=labels

Step outputs:
  candidate                 quality     $/min         $/mo       cost   latency  flags
  reference-v1                1.000       n/a          n/a    unknown   unknown  baseline, WINNER
  candidate-v3                0.975       n/a          n/a    unknown   unknown
  candidate-v2                0.900       n/a          n/a    unknown   unknown
  candidate-v2-partial        0.875       n/a          n/a    unknown   unknown  1 errors

Check the reference run against itself. This is the control: the policy in policy.json asks for 40 cases, no errors, no ungraded cases, a declared configuration, exact match at least 0.95, and a regression of at most 0.05.

$ redcrown check evidence.json --candidate-id reference-v1 --baseline evidence.json --baseline-candidate-id reference-v1 --policy policy.json
redcrown check: pass (exit 0)
- minimum_items: pass [requirement_met]; observed=40, threshold=40, count=40, denominator=40
- maximum_error_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- maximum_ungraded_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- require_known_config: pass [requirement_met]; observed=True, threshold=True, count=1, denominator=1
- quality.minimum: pass [requirement_met]; observed=1.0, threshold=0.95, count=40, denominator=40
- quality.maximum_regression: pass [requirement_met]; observed=-0.0, threshold=0.05, count=40, denominator=40

Check the candidate. It did not meet two of the six requirements, and the exit code says so.

$ redcrown check evidence.json --candidate-id candidate-v2 --baseline evidence.json --baseline-candidate-id reference-v1 --policy policy.json
redcrown check: fail (exit 2)
- minimum_items: pass [requirement_met]; observed=40, threshold=40, count=40, denominator=40
- maximum_error_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- maximum_ungraded_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- require_known_config: pass [requirement_met]; observed=True, threshold=True, count=1, denominator=1
- quality.minimum: fail [threshold_breached]; observed=0.9, threshold=0.95, count=40, denominator=40
- quality.maximum_regression: fail [threshold_breached]; observed=0.1, threshold=0.05, count=40, denominator=40

List the cases that changed. The per-case scores are in the evidence file; the script reads them and computes nothing new.

$ python3 changed_cases.py evidence.json reference-v1 candidate-v2
case      reference   reference-v1          candidate-v2          status
case-07   account     account               billing               regressed
case-18   technical   technical             account               regressed
case-25   billing     billing               technical             regressed
case-36   shipping    shipping              account               regressed
4 case(s) changed, 0 excluded, 40 in the set

Three more shapes the same files show:

• --candidate-id candidate-v3 passes (0.975, regression 0.025, exit 0).
• --candidate-id candidate-v2-partial fails on maximum_error_rate (one case timed out). The two quality rules are indeterminate with the reason incomplete_quality_coverage, count 39 of 40. A missing score is never a pass and never a zero.
• Score other-feature.json to other-evidence.json and check it against evidence.json: exit 2 with the evidence issues workload_mismatch and content_hash_mismatch. A different feature cannot be compared with this reference run.

check exits 0 on pass, 2 on fail or indeterminate, and 1 on a command, file or policy error (redcrown check policy.json --policy policy.json is the error case: candidate_raw_report_required). --format json prints the same decision with every rule's id, status, reason, observed, threshold, count and denominator.

Score outputs you already have (no model calls)

If another harness produced the outputs, score them locally with a deterministic metric (exact_match, similarity, json_field_accuracy, f1). This command never connects to a provider or needs an API key:

redcrown score score-request.json --report-json scored.json

score-request.json uses items (item_id, input, reference) and candidates (id, label, configuration, outputs) from the ScoreOutputsRequest schema. For tabular workflows, keep the items in CSV and put one candidate output per JSONL line:

redcrown score --items-csv items.csv --outputs-jsonl outputs.jsonl \
  --metric exact_match --quality-bar 0.8 --report-json scored.json

{"candidate_id":"model-a","item_id":"row1","output":"billing","latency_ms":42}

CSV accepts item_id (or id), input, and reference. JSONL also accepts candidate_label, error, cost_usd, and latency_ms. Omitted cost or latency stays unknown rather than being treated as zero. The resulting file can be sent to RedCrown with redcrown push scored.json.

The CSV+JSONL form is scorable but not checkable against a reference run: it carries no declared configuration, so redcrown check returns missing_or_invalid_config and every rule is indeterminate. The request JSON form with a configuration per candidate is the one check accepts.

Check prompt and model changes in CI

Export private evidence from each score, then check the candidate against the reference run and a policy:

redcrown score baseline-request.json --evidence-json baseline-evidence.json
redcrown score candidate-request.json --evidence-json candidate-evidence.json
redcrown check candidate-evidence.json --baseline baseline-evidence.json \
  --policy policy.json --format json

Use the same nonblank experiment_id and the same evaluation items and references in both score requests. Each selected candidate needs a complete configuration declaration, for example:

{
  "experiment_id": "support-routing",
  "quality_metric": "exact_match",
  "items": [{"item_id": "one", "input": "Refund request", "reference": "billing"}],
  "candidates": [{
    "id": "external-a",
    "configuration": {
      "provider": "external-harness",
      "model": "routing-model-v1",
      "prompt": "Return the routing category.",
      "system_prompt": null,
      "params": {},
      "source_revision": null,
      "endpoint_revision": null
    },
    "outputs": [{"item_id": "one", "output": "billing"}]
  }]
}

All seven configuration fields are required; null declares an absent setting, not an unknown one. RedCrown scores these supplied outputs but does not witness their generation. Configuration assurance remains declared. Keep secrets out of declarations and evidence files.

Example policy.json (choose thresholds for your own requirements):

{
  "schema_version": 1,
  "requirements": {"quality": {"minimum": 0.85, "maximum_regression": 0.01}}
}

When a report has multiple candidates, specify --candidate-id and --baseline-candidate-id; checks never select a winner automatically. Quality rules require complete scored coverage. Missing or incompatible evidence cannot silently pass. JSON is the supported policy format; YAML is deferred. Cost and latency rules parse but stay indeterminate until measurement provenance is supported.

check makes no network, provider, or authentication calls. JSON, Markdown, and text results describe the same decision. eval --evidence-json also exports private raw evidence; real evaluations may call providers. Keep a stable id in the experiment across those evaluations. --report-json remains the upload format for push; it is different from the private raw format consumed by check. Keep private evidence out of public CI artifacts. Select baseline and policy files from a trusted source.

Compare harness runs

You already keep a journal of what your AI feature did on a fixed set of cases (one JSONL row per case with a status and named checks, each ok, fail or unmeasured). RedCrown reads it, compares two runs check by check, and exports a payload for the record.

redcrown harness report <run> --runs runs/
redcrown harness compare <base-run> <head-run> --runs runs/     # exit 1 on any regression, 2 on corpus drift
redcrown harness export <run> --runs runs/ --name "one-button" --label "one-button@<commit>" --out payload.json
redcrown push payload.json

A status the harness wrote is authoritative. An unmeasured check is never a pass or a fail; a case with nothing measured is BLOCKED and is disclosed on the record, not counted as a failure. The two denominators are printed side by side. Coding agents: redcrown/harness/AGENTS.md ships in this package with the exact commands, exit codes and what never to claim.

redcrown matrix-plan previews explicit model and prompt axes offline and can export a new experiment file with --output. It never runs the experiment. The hosted matrix workspace, regression diagnosis and continuous evaluation are server features behind their own flags and are not part of this package.

Bundled sample (no keys needed)

Try the bundled offline demo, no provider keys required:

redcrown eval --sample extract

This runs a small bundled extraction sample using stub outputs. The output says demo mode because no real models are called. Connect your provider keys in ~/.redcrown/credentials.json (or via redcrown login) for real quality scores.

Bring your own data (text tasks)

If you have a CSV or JSONL file of inputs, import it directly:

redcrown build-dataset --from-csv mydata.csv --task classify --out exp.json
redcrown eval exp.json --report-json out.json

CSV/JSONL columns:

--task values: extract, classify, summarize, qa

The importer fills in the provider scaffold for you. Edit exp.json to swap in the candidates you want to compare before running.

Running a full evaluation

Model comparison is the supporting capability: the same cases across several models or configurations, ranked on the quality bar you set.

# run the fan-out locally, on your machine, with your keys
redcrown eval exp.json --report-json out.json

# (optional) sign in once per machine via device-code OAuth
redcrown login

# (optional) push the results to a shareable, no-login report page
redcrown push out.json --proof-link

redcrown eval ranks every config on cost, quality, and latency against your own ground truth and names the cheapest one that clears your quality bar. Example:

RANKED  transcription · cheapest config at or above your 0.85 quality bar
  deepgram · nova-3-medical    quality 0.883    $294/mo   winner, 40% cheaper
  aws · transcribe-standard    quality 0.879    $487/mo   incumbent
  openai · whisper-1           quality 0.820    $122/mo   below your bar

That run is published as a live, no-login report page: https://app.redcrown.ai/proof/O9iYVdWuaYjaL6mnImeIsD6TB1W_S6h4Frbx04YqAYQ

Note on build-dataset primock57: this command downloads the PriMock57 clinical-transcription corpus, which requires git-lfs. It is an audio eval designed for transcription benchmarking, not a general first step. Start with the regression quickstart above, redcrown eval --sample extract, or build-dataset --from-csv instead.

The exp.json schema

Every eval is a plain JSON file. Here is a minimal 2-candidate text example you can copy, edit, and run:

{
  "name": "classify-support-tickets",
  "quality_metric": "similarity",
  "quality_bar": 0.75,
  "reference_source": "labels",
  "pipeline": [
    {
      "id": "step1",
      "order": 1,
      "name": "classify",
      "input_type": "text",
      "output_type": "text",
      "input_binding": "dataset",
      "incumbent_candidate_id": "gpt4o-mini",
      "candidates": [
        {
          "id": "gpt4o-mini",
          "provider": "openai",
          "model": "gpt-4o-mini",
          "label": "GPT-4o Mini (incumbent)"
        },
        {
          "id": "gpt-oss-20b",
          "provider": "groq",
          "model": "openai/gpt-oss-20b",
          "label": "GPT-OSS 20B via Groq"
        }
      ]
    }
  ],
  "dataset": [
    {
      "id": "row1",
      "payload": { "kind": "text", "text": "My invoice is wrong." },
      "reference": "billing"
    },
    {
      "id": "row2",
      "payload": { "kind": "text", "text": "App crashes on login." },
      "reference": "bug"
    }
  ]
}

Required fields: name, quality_metric, quality_bar, reference_source, pipeline[].incumbent_candidate_id, pipeline[].candidates[].id, pipeline[].candidates[].provider, pipeline[].candidates[].model, dataset[].id, dataset[].payload.text.

No labels? Use the incumbent as reference

If you do not have labeled ground truth, leave reference off your dataset rows and set reference_source: "incumbent":

{
  "reference_source": "incumbent",
  ...
  "dataset": [
    { "id": "row1", "payload": { "kind": "text", "text": "My invoice is wrong." } }
  ]
}

RedCrown scores every candidate against your current model's own output, so you only need to beat or match what you run today. This is the fastest path to a cost comparison when you have real traffic but no labeled examples.

Free by construction

Evaluations run on your machine with your own provider keys, so RedCrown never sees your raw data and the run costs you nothing beyond your own inference. Only the results you choose to push become a hosted report. --no-receipts keeps raw outputs local and uploads aggregates only.

Already ran an evaluation elsewhere?

You do not have to run anything through RedCrown to get a shared report. Take the results from an evaluation you already ran, as a JSON in the RedCrown results format, and push them:

redcrown push results.json --proof-link

You get the same ranked, receipted, shareable report. The fastest path, with no install, is the web app at https://app.redcrown.ai/upload. An aggregate import carries no case-level evidence unless the payload carried receipts.

Evaluate your own OpenAI-compatible endpoint locally

If you have a private or self-hosted API that speaks the OpenAI chat-completion format, point redcrown eval at it without ever uploading your endpoint secret:

export REDCROWN_CUSTOM_HTTP='{"base_url":"https://api.example.com/v1","auth_header_name":"Authorization","auth_value":"Bearer sk-...","body_style":"openai"}'
redcrown eval exp.json

Set a candidate in your exp.json with "provider": "custom_http" and any "model" string your endpoint recognizes. The JSON connection stays local; it is read directly from the env var at eval time and is never persisted or uploaded.

For coding agents (MCP)

Coding agents (Claude, Cursor, Codex) drive the model evaluation loop over the hosted MCP server at mcp.redcrown.ai: scaffold an experiment, run it, import results, review outputs, and create a share link. score, check, harness compare and baseline acceptance are CLI or app only; there is no MCP tool for them. The server is open source: https://github.com/RedCrown-ai/redcrown-mcp

Links

• Site: https://redcrown.ai
• App: https://app.redcrown.ai
• Quickstart: https://redcrown.ai/docs/quickstart/
• MCP server (open source): https://github.com/RedCrown-ai/redcrown-mcp

License

redcrown.harness, the harness, score, check, push and verify commands, and the pure scoring helpers they import: Apache-2.0. The hosted record (the server extra and the API modules): Proprietary, (c) RedCrown.ai. One wheel carries both until the open modules move to a public repository.