CrewScore

Find missing written safeguards before your AI assistant goes live.

An offline linter that checks an AI agent's system prompt against 23 published controls (injection defense, human approval, cost limits, stop conditions) and fails CI when a control you require is missing. No model, no API key, open rules.

Try it live in your browser, no install →

Quickstart · Watch the film · How it works · CI · Guardrails · Status and limits

Created and maintained by Sarosh Hussain, an independent open-source maintainer. Technical claims are grounded in the code, tests, and cited validation material.

Why it exists

A support assistant ships on Friday. Does its prompt say a human approves a refund? Does it say what to do with instructions hidden in a customer's message, or when to stop? Usually nobody checks, because checking means reading every prompt by hand, and runtime red-teaming is slow, paid and hard to reproduce.

We scanned 356 publicly collected agent prompts: 83 production-labeled prompts and 273 general-purpose prompts. Among the production-labeled subset, median coverage was 10 of 100. GPT-Store median: 0 of 100. Numbers → · Shareable card →

CrewScore makes the written part cheap to check. It reads prompt files, and SYSTEM_PROMPT / system_prompt string literals in .py, .ts and .js source, and reports which of 23 published controls are written down. It plans fixes without touching your file unless you pass --apply. In CI you name the controls your workflow needs, and the build fails only when one of those is missing.

Public-good thesis: written safeguards should be inspectable with an offline, open ruleset before teams depend on runtime claims that are harder to reproduce or independently review.

Quickstart (30 seconds)

pip install crewscore
mkdir crewscore-try && cd crewscore-try          # a fresh folder, so scan . sees only this prompt
mkdir -p prompts && echo "You are a friendly support assistant for an online store." > prompts/support.md

crewscore test --prompt-file prompts/support.md            # coverage N/23 and the first gap to review
crewscore fix --prompt-file prompts/support.md --plan      # which dimensions need wording; writes nothing
crewscore scan . --require human_gate.approval_required    # exit 2: the required control is missing

echo "A human must approve any refund or account change before it happens." >> prompts/support.md
crewscore scan . --require human_gate.approval_required    # exit 0: the control is written down

Python 3.11 or later. The CLI scores locally: no account, no network call, no LLM. Inside an existing project, scan . --require checks every prompt it finds, so use test --prompt-file <file> --require <control> to gate one file.

See it run

Watch the 90-second film → A one-line support prompt covers 0 of 23 controls. The release gate requires a human-approval rule and fails with exit 2. One sentence, written by a person, turns it to exit 0, and the film ends in the rule source that matched it.

How it works

flowchart LR
  SRC["Prompt files, AGENTS.md,<br/>SYSTEM_PROMPT literals in .py / .ts / .js"] --> KIND{"Artifact type<br/>by filename and path"}
  KIND -- "coding-agent config<br/>AGENTS.md, CLAUDE.md, .cursorrules" --> SMELL["Configuration smells<br/>never change the score"]
  KIND -- "agent system prompt" --> RULES["Open ruleset<br/>deterministic patterns + negation guards"]
  RULES --> CTRL["23 controls<br/>across 8 dimensions"]
  CTRL --> OUT1["coverage N/23,<br/>first gap, fix plan"]
  CTRL --> GATE{"--require control"}
  GATE -- missing --> FAIL(["exit 2: CI fails and names the control"])
  GATE -- written --> PASS(["exit 0"])
  CTRL --> CI["Prompt-free JSON, SARIF,<br/>sticky PR comment"]

Each rule is a deterministic pattern with guards, so "do not ask permission" never counts as a human-approval rule. Several rules roll up to one control, and the gate checks controls by name rather than an average. How scoring works → · Architecture →

Guardrails

• Offline, deterministic scoring. The CLI and Action score with no model, no API key and no network call, and the same prompt text always gets the same governance findings. Configuration-smell checks also read the file's Git history and neighboring linter config. The browser checker scores locally too; the site itself sends anonymous, opt-out usage telemetry, described in its privacy page.
• Writes nothing unless asked. fix --plan lists what is missing; only fix --apply writes template text, and a person decides what stays in the prompt.
• Gate on named controls, never the average. --require and required-controls fail only on the controls you name.
• CI output is prompt-free by default. Logs, JSON, Markdown and SARIF carry rule IDs and control labels, not the matched prompt text. The deprecated include-snippets: "true" input re-admits snippets to JSON and Markdown; SARIF stays prompt-free.
• Config files are not scored as prompts. AGENTS.md and similar files get a smell verdict, not a governance grade.
• Every scoring change is measured. The CHANGELOG records each change and its corpus delta.

Read this first

CrewScore is a checklist, not a benchmark. The number is the share of 23 published controls your prompt states — nothing about whether they are well specified, mutually consistent, or obeyed at runtime.

So a low score is actionable — you probably have not written down an injection policy, a human gate, or a safe-stop rule, and those are worth writing. A high score means the text is present, not that the agent obeys it. Don't rank prompts, teams, or vendors by this number, and don't treat a threshold as a safety bar. Prefer the findings to the total.

Three dimensions — Cost, Compliance, Audit — ship with known-thin construct validity and say so. Ruleset 0.6.0 tightened their patterns against measured corpus false positives; Compliance is still not lawful handling.

📄 The validation study → — including the arithmetic showing our own scale was broken through 0.1.0, which we published before fixing.

📊 Measured against 356 publicly collected prompts → — Cliff's δ = 0.614 separating production-labeled agent prompts from general-purpose ones, generated by a committed harness rather than typed by hand.

Usage

crewscore scan .                          # prompts + AGENTS.md + inline SYSTEM_PROMPT=...
crewscore scan . --no-inline              # file discovery only
crewscore init .                          # prompt-free regression baseline + PR workflow
crewscore scan . --fail-on-regression --baseline .crewscore-baseline.json
crewscore scan . --require human_gate.approval_required   # one-control CI habit
crewscore test --prompt-file ./prompt.md  # coverage N/23 + first gap to review
crewscore fix  --prompt-file ./prompt.md --plan   # what's missing, no writes
crewscore rules --concepts                # the 23 controls, and the rules behind them

Full CLI reference → · How scoring works →

CI

Start report-only — it never fails a build:

- uses: shmindmaster/crewscore@v2
  with:
    scan-path: "."

When you know which controls your workflow actually needs, name them — the build then fails only when a named control is missing, never on the coverage average:

- uses: shmindmaster/crewscore@v2
  with:
    scan-path: "."
    required-controls: "human_gate.approval_required,safe_stop.stop_condition"
    sarif: "crewscore.sarif"

Posts a sticky PR comment with the open rule findings (give the job permissions: pull-requests: write; without it the action logs a warning and moves on). Failed gates surface as ::error annotations naming the control.

CI output is prompt-free by default: rule IDs, dimensions, status, and control labels — never the matched text of the prompt that was scanned, because that build log is more public than the prompt is. include-snippets: "true" re-admits it to JSON and Markdown output and is deprecated; it will be removed after one release. SARIF remains prompt-free regardless of that input. Guard downstream steps on the scored output, not on score — an empty score casts to 0.

Action inputs, outputs, and the CLI variant →

Two artifacts, two rulesets

CrewScore judges two kinds of file, and tells you which it thinks it is looking at. Detection is by filename and path — never by sniffing content.

A file saying "always use pnpm" is telling a coding agent how to work in your repo. It has no reason to contain HIPAA language, and scoring it against that is a category error.

We know the size of that error because we measured it: against the 100 most-starred repos with an AGENTS.md (arXiv:2606.15828), the governance ruleset put all 100 in the worst tier. A scale the entire population fails carries no information. So config files get a smell verdict instead — and in --json, no governance grade at all.

crewscore test --prompt-file AGENTS.md
# -> CONFIG: NO SMELLS DETECTED   (not "0/100 CRITICAL GAPS")

Configuration smells

Problems in the shape of an instruction file rather than its content, from a published catalog — Configuration Smells in AGENTS.md Files (dos Santos et al., 2026), which found 91 of 100 popular projects carried at least one.

The paper's other three smells need an LLM to detect. We would rather ship three honest detectors than six approximate ones. Lint Leakage is an approximation of the paper's detector and says so in its output; Init Fossilization cannot tell "never needed revising" from "never got revised."

Smells never change the score. Folding them in would silently change what every existing --threshold means.

Docs

Status and limits

Current release: see PyPI and the CHANGELOG. Ruleset 0.6.0. The GitHub Action is shmindmaster/crewscore@v2, and the checker at crewscore.ai runs the same engine locally in your browser. The project is Alpha: the JSON payload and exit codes can still change between releases, and every change is recorded.

What this is not: live adversarial red-teaming, runtime tool-gate enforcement, a security or compliance certification, or proof that the model will obey the text. Three dimensions (Cost, Compliance, Audit) ship with disclosed thin construct validity. For runtime testing after the structural gate, see the live eval handoff to Promptfoo or garak.

Roadmap: framework adapters that extract prompts from LangGraph, CrewAI and AutoGen graphs, and optional live adversarial testing (not the default path). See the roadmap.

Contributing, security and license

• Contributing: read CONTRIBUTING.md and the development guide. A new rule needs a corpus measurement and a negative test. Good first issues are labelled good first issue; questions go to Discussions.

  git clone https://github.com/shmindmaster/crewscore.git
  cd crewscore
  pip install -e ".[dev]"
  pytest
  

• Security: report vulnerabilities privately through a GitHub security advisory, as described in SECURITY.md.

• License: MIT · Governance · Roadmap

Related open source by the same maintainer: Assay keeps an agent's decision out of its side effects, and GitPin makes agent-authored pull requests carry exact, re-checkable evidence. More at saroshhussain.com.