crewscore 0.6.0

Find the guardrails your AI agent prompt forgot: which of eight failure modes - prompt injection, hallucination, runaway cost, missing human approval and more - the text never guards against. Offline, deterministic, explainable per-rule findings, with a CI gate.

CrewScore

Find the safety rules your AI agent prompt forgot.

CrewScore reads your system prompt and shows you which of 23 safety controls it never states — injection defense, human approval, cost limits, stop conditions.

We scanned 356 real agent prompts: 83 production prompts and 273 general-purpose prompts. Among the production subset, median coverage was 14 of 100. See the numbers →

Example result: 8 of 23 written guardrails found · 15 may be missing. CrewScore checks whether controls are written down, not whether an agent obeys them.

Try it live, no install: crewscore.ai

pip install crewscore
crewscore scan .

Deterministic regex over prompt text. Offline, no API key, no LLM.

---

Read this first

CrewScore is a checklist, not a benchmark. The number is the share of 23 published controls your prompt states — nothing about whether they are well specified, mutually consistent, or obeyed at runtime.

What the prompt does	What it scores
Nothing written down	0
One control in each of the 8 dimensions	36
All 23 controls	100
One control restated five different ways	same as stating it once

So a low score is actionable — you probably have not written down an injection policy, a human gate, or a safe-stop rule, and those are worth writing. A high score means the text is present, not that the agent obeys it. Don't rank prompts, teams, or vendors by this number, and don't treat a threshold as a safety bar. Prefer the findings to the total.

Three dimensions — Cost, Compliance, Audit — ship with known-poor construct validity and say so. Compliance is keyword detection; naming a regulation is not complying with it.

📄 The validation study → — including the arithmetic showing our own scale was broken through 0.1.0, which we published before fixing.

📊 Measured against 356 real prompts → — Cliff's δ = 0.672 separating production agent prompts from general-purpose ones, generated by a committed harness rather than typed by hand.

---

Usage

crewscore scan .                          # find and score every agent prompt
crewscore init .                          # create a prompt-free regression baseline + PR workflow
crewscore scan . --fail-on-regression --baseline .crewscore-baseline.json
crewscore scan . --require human_gate.approval_required
crewscore test --prompt-file ./prompt.md  # score one file
crewscore fix  --prompt-file ./prompt.md --plan   # what's missing, no writes
crewscore rules --concepts                # the 23 controls, and the rules behind them

Full CLI reference → · How scoring works →

---

CI

- uses: shmindmaster/crewscore@v2
  with:
    scan-path: "."
    # Report-only by default. Protect controls explicitly instead of treating
    # the coverage average as a safety bar:
    required-controls: "human_gate.approval_required,safe_stop.stop_condition"
    sarif: "crewscore.sarif"

Posts a sticky PR comment with the open rule findings. Guard downstream steps on the scored output, not on score — an empty score casts to 0.

Action inputs, outputs, and the CLI variant →

---

Two artifacts, two rulesets

CrewScore judges two kinds of file, and tells you which it thinks it is looking at. Detection is by filename and path — never by sniffing content.

Artifact	Examples	Judged on
Coding-agent config	AGENTS.md, CLAUDE.md, .cursorrules	Configuration smells
Agent system prompt	system-prompt.md, anything under prompts/ or agents/	The 8 governance dimensions

A file saying "always use pnpm" is telling a coding agent how to work in your repo. It has no reason to contain HIPAA language, and scoring it against that is a category error.

We know the size of that error because we measured it: against the 100 most-starred repos with an AGENTS.md (arXiv:2606.15828), the governance ruleset put all 100 in the worst tier. A scale the entire population fails carries no information. So config files get a smell verdict instead — and in --json, no governance grade at all.

crewscore test --prompt-file AGENTS.md
# -> CONFIG: NO SMELLS DETECTED   (not "0/100 CRITICAL GAPS")

---

Configuration smells

Problems in the shape of an instruction file rather than its content, from a published catalog — Configuration Smells in AGENTS.md Files (dos Santos et al., 2026), which found 91 of 100 popular projects carried at least one.

Smell	Heuristic	Found in
Context Bloat	≥ 200 lines	42% of studied projects
Lint Leakage	Style rules a configured linter already enforces	62%
Init Fossilization	Tracked by git with exactly one commit	24%

The paper's other three smells need an LLM to detect. We would rather ship three honest detectors than six approximate ones. Lint Leakage is an approximation of the paper's detector and says so in its output; Init Fossilization cannot tell "never needed revising" from "never got revised."

Smells never change the score. Folding them in would silently change what every existing --threshold means.

---

Development

git clone https://github.com/shmindmaster/crewscore.git
cd crewscore
pip install -e ".[dev]"
pytest

AGENTS.md has contributor operating notes. CONTRIBUTING.md covers adding a rule — note that adding a synonym for a control that is already covered changes no score, by design.

---

Docs

Validation	What the number does and does not measure
Corpus validation	Generated result over 356 real prompts
Scoring	Formula, dimensions, provenance, charter
CLI	Every command and flag
CI	Action and CLI integration
Policies and SARIF	Regression and required-control CI without score gating
Architecture	One catalog powering CLI, Action, and browser
Scoring governance	How public rules and validation change
Roadmap	Available work and deliberately deferred capabilities
Security	Private vulnerability reporting
Comparison	Other tools, and what to use after this one
CHANGELOG	Including every scoring change and its measured delta

---

What this is not

Live adversarial red-teaming · runtime tool-gate enforcement · a security or compliance certification · proof the model will obey the text.

Roadmap: framework adapters that extract prompts from LangGraph / CrewAI / AutoGen graphs; optional live adversarial testing (post-traction, not the default path).

MIT licensed.