sigmaforge 0.1.0

Honest Sigma-rule backtest harness

sigmaforge

Honest Sigma-rule backtest harness. Measures detection rules against real log corpora and reports two numbers per rule — recall (does it catch attacks of its ATT&CK sub-technique?) and precision / false-positives (does it fire on benign activity?) — with honesty gates that return unmeasured instead of a fake 0 or a tautological 1.0 when the data can't support a number.

[!NOTE] Learning / portfolio project, built by directing AI coding agents. Christian Huhn (photography → SOC career change) designed, reviewed, and gated the work; the implementation was AI-pair-programmed. It is an honest measurement harness, not a polished product — see Status below for exactly what works and what doesn't yet.

What problem it solves

Every SOC ships dozens to hundreds of detection rules and rarely measures them. sigmaforge answers two questions with reproducible evidence:

• Which rules are noise generators? (high false-positives on legitimate activity)
• Which rules catch nothing? (zero recall against real attacks of their technique)

Example finding from a real run: Suspicious Windows Service Tampering produced 66 false-positives on a benign corpus — every one a Ninite / TeamViewer installer, not an attack.

How it actually works

flowchart LR
    R[SigmaHQ rules] -->|partition high/critical| C[compile to one Zircolite ruleset]
    C --> E[Zircolite engine]
    A[(attack corpus<br/>sub-technique-labeled)] --> E
    B[(benign corpus<br/>Nextron + OpTC)] --> E
    E --> S[score: recall per technique<br/>+ precision/FP label-aware]
    S --> G[honesty gates<br/>floor · positive-control · no-self-review]
    G --> O[report.md + manifest]

The real pipeline is script-driven (scripts/run6_backtest.py is the current end-to-end path):

uv run python scripts/compile_loaded_ruleset.py   # rules -> one Zircolite ruleset
uv run python scripts/run6_backtest.py            # backtest -> reports/run6.md

[!WARNING] The shipped CLI (sigmaforge backtest) is a weaker, work-in-progress path and is not the way the real reports were produced. Use the scripts above. The CLI is kept for the future one-command experience, not parity.

Status

Area	State
Recall (per sub-technique, no sibling dilution)	Working — 338/609 rules measurable, 70 fire (run5)
Precision / false-positives (label-aware, gated)	Working — 7/609 measurable on current benign corpus (run6)
Honesty gates (floor, positive-control, no-self-review)	Working
Reproducible manifest (run_hash, corpus SHAs, provenance)	Working
One-command CLI (sigmaforge backtest)	WIP — weaker than the scripts
Self-generated benign corpus	Kit ready (scripts/selfgen/), needs a Windows VM run

[!IMPORTANT] The log corpora are not shipped. They are large, separately licensed, and gitignored. pip install sigmaforge installs the harness code, not the data — a full end-to-end backtest needs the corpora and a local Zircolite checkout (also not bundled). The package is useful as a library / reference; the runnable pipeline needs the local setup documented in scripts/.

Install

pip install sigmaforge

Installs the harness package and the sigmaforge CLI. The detection engine (Zircolite) and the log corpora are obtained separately (see above).

Corpora used (all verified, portfolio-safe licenses)

Corpus	Role	License
splunk/attack_data	recall (sub-technique-labeled attacks)	Apache-2.0
DARPA OpTC	precision (real enterprise benign week)	Public domain
NextronSystems/evtx-baseline	precision (goodware baseline)	Apache-2.0
Self-generated (scripts/selfgen/)	precision (targeted admin/LOLBin noise)	your own lab

Development

Built with the Shipwright dev framework.

uv sync --dev
uv run pytest        # 108 tests
uv run ruff check .

License

MIT © Christian Huhn. Corpus data retains its upstream license (see table above).