crewai-no-vibes 0.1.0

CrewAI Task guardrail blocking verification claims without evidence — empirical F1 0.815 on MAST mode 3.3

crewai-no-vibes

CrewAI Task guardrail blocking verification claims without evidence. Pure-Python, zero runtime dependencies, drop-in.

Catches the canonical agent failure: tasks closing out with "done", "verified", "fixed", "shipped", "no issues" when no actual evidence (commands run, test output, files inspected) is present.

Empirical baseline

no-vibes is the strongest measured detector for MAST mode 3.3 ("No or Incorrect Verification") — the highest-prevalence multi-agent failure mode in the canonical taxonomy.

Dataset	n	F1	95% CI	Precision	Recall
MAD human-labelled	19	0.815	[0.615, 0.941]	0.733	0.917
MAD LLM-judge full	954	0.308	[0.264, 0.352]	0.226	0.486

Inter-annotator Fleiss κ for mode 3.3 on the released human-labelled set: 1.000 (perfect agreement). Bootstrap: percentile method, B=10000 resamples, seed=42.

Full empirical report with methodology, per-MAS-framework breakdown, and limitations: https://github.com/waitdeadai/llm-dark-patterns/blob/main/evaluation/MAST-RESULTS.md

Install

pip install crewai-no-vibes

Or directly from this repository:

pip install git+https://github.com/waitdeadai/crewai-no-vibes.git

Pure Python, supports 3.9+, no external binary required. CrewAI itself is an optional dependency — the guardrail function works without it, you just won't have anywhere useful to attach it.

Usage

from crewai import Agent, Task
from crewai_no_vibes import verification_claim_evidence_guardrail

researcher = Agent(role="Researcher", goal="...", backstory="...")

task = Task(
    description="Run the benchmark and report results.",
    expected_output="A summary with command output and verification status.",
    agent=researcher,
    guardrail=verification_claim_evidence_guardrail,
)

When the agent's output claims completion without evidence, the guardrail returns (False, feedback) and CrewAI retries the task with the feedback as context. CrewAI's max_retries (default 3) caps the retry loop.

What it catches

The guardrail blocks outputs that trigger done_without_evidence:

claims_completion AND (NOT has_evidence_marker OR completion_with_negative_evidence)

Where:

• claims_completion: message contains any of done, complete, completed, ready, implemented, fixed, resolved, verified, deployed, shipped, all set, no issues
• has_evidence_marker: message contains command evidence (e.g. pytest, cargo test, commands run: ...) OR verification-pass evidence (verification: passed, tests passed) OR read-only-audit evidence (files inspected:, sources reviewed:)
• completion_with_negative_evidence: completion claim PLUS a negative evidence marker (verification: not run, unverified, tests not run, could not verify) — i.e. "I'm done" while explicitly admitting verification didn't happen

Catches

"Done. All set." → BLOCKED
"Fixed. No issues found." → BLOCKED
"Implemented and ready, with no observable command or file evidence." → BLOCKED
"All tests passed, verified. (Tests not run.)" → BLOCKED  # negative-evidence override

Passes

"Implemented. Commands run: `pytest tests/` → 8 passed. Verification: passed." → OK
"Read-only audit complete. Files inspected: src/parser.py, tests/test_parser.py. No changes." → OK
"Status: partial. The scaffolding is drafted. Verification: not run because no fixture exists yet. Next step: write fixture." → OK  # honest partial with no completion claim
"Reviewing the design — no decision yet." → OK

Configuration

The default guardrail uses the upstream rule pack's settings. For customization, use the factory:

from crewai_no_vibes import make_guardrail

# Custom feedback message
guardrail = make_guardrail(
    custom_feedback="Show test output, please."
)

# Stricter mode: block on ANY completion claim without command/verification evidence,
# regardless of whether a negative-evidence marker is present.
strict_guardrail = make_guardrail(require_command_or_verification=True)

Source ledger

This package is a pure-Python port of the evidence_claims rule pack from agent-closeout-bench, the Rust YAML rule pack engine maintained alongside llm-dark-patterns.

• Rule pack source: rules/closeout/evidence_claims.yaml
• Engine source: engine/src/main.rs extract_features + helpers
• Engine sha256 at port time: ffe3c4e5dce01505...
• Rule pack hash at port time: sha256:26fa8fd9999c055d...

If the upstream rule pack updates, this package version-pins to its own snapshot. Re-port for new versions; do not assume drift-free behavior.

Honest limitations

1. Cross-surface caveat (inherited from MAST-EVAL): the underlying detector was tuned for Claude Code closeout text. MAD is multi-agent trajectory text. The F1 0.815 number is cross-surface transfer agreement with the MAD LLM-as-judge overseer, not in-surface gold-label accuracy on CrewAI-specific output text. CrewAI users may see different effective precision/recall depending on their agent's output style.
2. Pure-Python port may diverge from Rust engine on edge cases: regex flavor (Rust regex crate vs Python re) and Unicode handling can differ. Tests cover the rule pack's documented examples and core helper predicates; if you find a divergence, please open an issue with the input that exposes it.
3. No LLM-judge fallback: this is the deterministic floor. For semantic-reasoning catches the rule pack misses, layer an LLM-based guardrail downstream (CrewAI also supports those via string-description guardrails).
4. Retry loop bounded: CrewAI's max_retries (default 3) caps retries. If the agent persistently outputs done_without_evidence, the loop exits with the latest output despite the guardrail blocks. Set max_retries and the task's description accordingly.

Related work

• MAST taxonomy (Cemri et al., NeurIPS 2025): https://arxiv.org/abs/2503.13657 — peer-reviewed catalogue of 14 multi-agent failure modes. Mode 3.3 is what this guardrail targets.
• ARCF (Arora & Singh, IJERT 2026): https://www.ijert.org/a-reliability-control-framework-for-robust-multi-agent-llm-systems-managing-workflows-in-large-language-model-systems-ijertv15is050114 — real-time MAS reliability framework citing MAST; flags rule-based detection as a limitation. This package is one empirical data point on what rule-based / deterministic detection can achieve.
• DarkBench (Kran et al., ICLR 2025): https://arxiv.org/abs/2503.10728 — dark pattern corpus; complementary chat-surface evaluation.
• CrewAI Task guardrails docs: https://docs.crewai.com/en/concepts/tasks
• CrewAI quickstart for guardrails: https://github.com/crewAIInc/crewAI-quickstarts/blob/main/Guardrails/task_guardrails.ipynb
• CrewAI PR #1742 (feature that introduced task guardrails): https://github.com/crewAIInc/crewAI/pull/1742

Development

git clone https://github.com/waitdeadai/crewai-no-vibes.git
cd crewai-no-vibes
pip install -e ".[dev]"
pytest

Or without installing (system Python with PEP 668 protection):

PYTHONPATH=src pytest tests/

License

Apache-2.0. See LICENSE.

Author

Fernando Lazzarin · waitdead.com · restlessmachine.com