audit-test 0.4.1

Code and test verification orchestrator — interrogates code quality AND whether tests genuinely prove correctness

🐇 audit-test

One command. One policy. One fail-closed verdict.

Polyglot code auditor — syntax, wiring, PHD static analysis, runtime checks, test suite health, quality gates, deep tree-sitter AST rules, and 23 native linters across 21 languages. All in one shot. Zero config needed.

Install

pip install audit-test

This pulls the core Python linters automatically (ruff, bandit, mypy, black, coverage) plus the tree-sitter grammars behind the deep polyglot AST rules (JS/TS, Rust, Go, Java, C#, Kotlin, Swift, PHP) — so the Python audit and the deep multi-language rules work out of the box, no separate setup.

Want the heavier / niche scanners too (semgrep, pip-audit CVE scan, mutmut mutation testing, sqlfluff)?

pip install audit-test[all]

Non-Python linters (ESLint, Prettier, clippy, go vet, clang-tidy, …) are external toolchains pip can't install. audit-test detects them at runtime and cleanly SKIPs any that are missing — it never fakes a pass.

Or from source:

git clone https://github.com/Yosef-Bunick/AUDIT-TEST.git
cd AUDIT-TEST
pip install -e .          # or  pip install -e ".[all]"

Usage

Three commands — all identical:

audit-test
audit-code
audit-tests

Bare words or flags — both work:

audit-test                     # full audit
audit-test v                   # full audit, verbose
audit-test f                   # fix: auto-format (lint+black)
audit-test min                 # min: fast wiring + phd + quality
audit-test F                   # full: checks + raw output
audit-test w r h               # wiring+runtime high only
audit-test -p <dir>            # audit a specific project
audit-test -s "s q"            # skip whats in next value "suite + quality"

Quick keys

Key	Module	Runs
p	phd	exception discipline, security patterns, state bugs
w	wiring	dead symbols, test-only code, config drift
r	runtime	unbounded loops, timeouts, secrets in logs
s	suite	pytest, solo reruns, HEAD diff baseline
q	quality	black, ruff, mypy, CVE, coverage
l	lint	ruff lint
b	black	black format
d	deps	dependency scanner

Mode shortcuts:

Key	Does
h	HIGH severity (default)
m	HIGH + MEDIUM severity
v	verbose output
f	auto-format (~1s)
F	full analysis
fast	skip slow checks

Full keyword reference

All forms work — bare words, -short, or --long:

Bare	Short	Long	Does
Severity
high	-h	--high	HIGH only (default)
medium	-m	--medium	HIGH + MEDIUM
info		--info	HIGH + MEDIUM + INFO
all		--all	all findings
Modes
fix	-f	--fix	auto-format (quality)
full	-F	--full	complete analysis
fast		--fast	skip slow checks
verbose	-v	--verbose	full detail output
min		--min	fast: wiring + phd + quality + deps
strict		--strict	exit non-zero on any FAIL/CRASH (default)
report		--report-only	print findings but always exit 0
Options
	-p	--path PATH	project directory
skip	-s	--skip MODULES	skip modules
		--json FILE	write JSON report to FILE
		--sarif FILE	write SARIF report (GitHub code scanning)
		--junit FILE	write JUnit XML report to FILE
		--profile NAME	enable a project-specific audit profile
		--config FILE	path to audit-code.toml
	-H	--help	show help
Core modules
phd p		--phd	PHD static audit
wiring w		--wiring	wiring audit
runtime r		--runtime	runtime audit
suite s		--suite	test suite audit
quality q		--quality	quality gates
encoding		--encoding	source-encoding check
syntax		--syntax	all language syntax checks
python		--python	Python syntax only
tests		--tests	non-Python test suites
lint l		--lint	ruff lint
black b		--black	black format
deps d		--deps	dependency scanner
Security integrations
semgrep		--semgrep	semgrep (structural)
bandit		--bandit	bandit (Python security)
megalinter		--megalinter	MegaLinter umbrella scan (opt-in, slow)
Language integrations — SKIP if tool not installed
eslint		--eslint	ESLint (JS/TS)
prettier		--prettier	Prettier (JS/TS/CSS)
checkstyle		--checkstyle	Checkstyle (Java)
pmd		--pmd	PMD (Java)
go-vet		--go-vet	go vet (Go)
golangci-lint		--golangci-lint	golangci-lint (Go)
clippy		--clippy	cargo clippy (Rust)
rustfmt		--rustfmt	rustfmt (Rust)
dotnet-format		--dotnet-format	dotnet format (C#)
clang-tidy		--clang-tidy	clang-tidy (C++)
cppcheck		--cppcheck	cppcheck (C++)
htmlhint		--htmlhint	HTMLHint (HTML)
stylelint		--stylelint	Stylelint (CSS/SCSS)

The newer per-language linters — PHPStan, RuboCop, SwiftLint, detekt, dart analyze, Scalafix, Credo, zig fmt, Luacheck, HLint — have no dedicated flag: they run automatically whenever their language is detected in the tree, and SKIP cleanly when the tool isn't installed.

Focus groups

Save file sets in .audit-test-ignore and run audits against them:

audit-test focus add fast main.py cli.py   # create group
audit-test focus info                        # list all groups
audit-test focus fast                        # audit the 'fast' group
audit-test focus fast v /mnt/c/other         # verbose, from other path
audit-test focus path fast /mnt/c/other      # set group path
audit-test focus desc fast "quick checks"    # set description
audit-test focus del fast cli.py             # remove file from group
audit-test focus clear fast                  # delete group

Ignore patterns

Manage .audit-test-ignore skip patterns directly:

audit-test ignore add generated/             # add skip pattern
audit-test ignore info                       # list patterns
audit-test ignore del generated/             # remove pattern
audit-test ignore clear                      # remove all custom patterns

Encoding check

Verify every text file decodes cleanly under a chosen encoding (strict — no silent replacement). Binary files are skipped automatically.

audit-test check utf-8         # every file must be valid UTF-8
audit-test check ascii         # pure ASCII only
audit-test check UTF-16        # names are case/space-insensitive
audit-test check GB 18030      # multi-word names → gb18030
audit-test check               # use the project's configured encoding
audit-test check utf-8 -p <dir>   # check another project

Set a project's expected encoding once in its .audit-test-ignore — then a bare check (or a scan of that project) uses it:

#encoding utf-8

Precedence: explicit argument → the target's #encoding → utf-8. Exit code is 0 when every file matches, 1 when any file doesn't.

The check also runs automatically inside a full/default audit (using the configured encoding) — so audit-test full already includes it. Use the standalone check command when you want to test a specific encoding on demand.

check utf-8 / ascii / gb18030 are strict. check utf-16 / utf-32 are weaker — those decoders accept almost any even-length byte run, so they mainly catch truncated files rather than proving the encoding.

Change gate

audit-test gate               # judge working-tree diff vs HEAD
audit-test gate high           # block on new HIGH findings (default)
audit-test gate medium         # block on new HIGH+MEDIUM
audit-test gate fast           # skip mutation (G4)
audit-test gate -p <dir>      # gate a specific project

Project profiling

Read and AST-parse every source file once to size up a project, or rank sibling projects side by side:

audit-test profile                    # structure/cost/features of the cwd
audit-test profile -p <dir>           # profile another project
audit-test profile --json out.json    # full profile as JSON

audit-test compare -p <root>          # rank every subproject under <root>
audit-test compare -p <root> --audit  # add wiring+phd HIGH counts (slower)
audit-test compare --skip a,b         # ignore named subdirs

profile reports structure (LOC, functions, classes, imports), parse cost, architecture (monolith vs modular + pipeline stages), runtime/performance signals, and — when configured — capability keyword buckets. compare lays that out as one row per subproject; --audit adds a HIGH-findings column.

Domain vocabulary is config-driven, never hardcoded. With no config only generic, language-level signals are reported. To specialise, add a [profile] table to audit-code.toml:

[profile]
pipeline_verbs  = ["render", "apply", "process"]
heavy_libs      = ["numpy", "cv2", "torch"]
utility_markers = ["lerp", "clamp"]
capabilities    = { products = ["foo", "bar"] }

Dead-symbol triage

Wiring tells you a symbol is dead; deadcode tells you whether it matters — sorting each into critical (a pipeline/feature function that never runs), utility (a harmless helper), or other (verify manually):

audit-test deadcode                   # classify dead symbols in the cwd
audit-test deadcode -p <dir>          # another project
audit-test deadcode --json out.json

Buckets are driven by the same [profile] pipeline_verbs / utility_markers config as profiling.

Surgeon — surgical & cross-file edits

Precise, line-numbered edits for agent workflows — no text matching, no escaping fragility. Every edit auto-runs black + ruff (--no-format to skip):

audit-test surgeon replace <file> <start>:<end> <content>      # overwrite lines
audit-test surgeon insert <file> <line> <content>              # insert after line
audit-test surgeon dry-run <file> <start>:<end> <content>      # preview without writing
audit-test surgeon batch <fixes.json>                          # apply many fixes
audit-test surgeon copy <src> <s>:<e> <dest> <after>           # copy lines across files
audit-test surgeon replace-cross <src> <s>:<e> <dest> <s>:<e>  # overwrite dest range with src
audit-test surgeon port <src> <dest> <function>                # move a function + its imports

port finds a function in <src> (a file or a project directory), copies it into <dest>, and brings along only the imports it actually references that <dest> is missing.

Context scanner

Quickly grab lines around a finding for context before fixing:

audit-test scan file.py 42               # ±3 lines around line 42
audit-test scan file.py 42 +5            # 5 lines after
audit-test scan file.py 42 -5            # 5 lines before
audit-test scan file.py 42 +5 -2         # 5 after, 2 before
audit-test scan file.py 15:30            # exact range
audit-test scan file.py 42 --json        # machine-readable output

Dependency graph

Trace import relationships — same vibe as scan:

audit-test graph cli.py              # ±2 steps (default)
audit-test graph cli.py +5            # 5 downstream (what it imports)
audit-test graph cli.py -3            # 3 upstream (what imports it)
audit-test graph cli.py +5 -3         # 5 forward, 3 back
audit-test graph cli.py +5 -3 --json  # machine-readable for agents

The graph walks 10 languages — Python, JS/TS, Rust, Go, Java, Kotlin, Swift, PHP, C#, and C/C++ — and also reports cross-language edges: subprocess calls and FFI bindings that jump from one language into another.

Speed

A full audit-test runs the test suite under coverage once: the suite audit produces the data and quality's Q5 (per-def execution proof) reuses it — no second run. Quality-only mode (audit-test q v) has no suite to share with, so Q5 caches the coverage keyed by a byte-fingerprint of every source and test file. Re-running q with unchanged code reuses that cache (≈20× faster); any edit busts the fingerprint and reruns. Force a fresh run with:

$env:AUDIT_NO_Q5_CACHE = "1"    # PowerShell — disable the Q5 coverage cache

Other fast paths: audit-test fast skips the slow checks (Q3 mypy, Q5 coverage, mutation); audit-test min runs only wiring + phd + quality. Test parallelism (pytest -n) is not used — this suite is subprocess-bound, so extra workers run slower, not faster.

Standalone scripts

The original audit scripts also work standalone — no pip install needed. Copy them into any project and run directly:

python audit_wiring.py         # dead symbols, config drift
python audit_wiring.py --dead-json dead.json   # + structured dead-symbol export
python audit_phd.py            # exception discipline, security patterns
python audit_phd.py --min-severity=HIGH   # HIGH findings only
python audit_runtime.py        # timeouts, log hygiene, prompt contracts
python audit_suite.py          # run pytest, classify failures
python audit_quality.py        # black, ruff, mypy, CVE, coverage
python audit_gate.py           # judge working-tree diff vs HEAD
python run_all_audits.py       # orchestrate all five into one report

The stack

Audit	Question it answers	Full rules
wiring	Is it connected? Dead symbols, test-only code, config key drift — framework-aware (FastAPI routes, Alembic migrations)	docs/wiring.md
phd	Does it meet the bar? Exception discipline, security patterns, state bugs	docs/phd.md — 55 rules
runtime	Will it hang or crash? Unbounded loops, missing timeouts, secrets in logs	docs/runtime.md — 13 checks
suite	Is the test suite healthy? Runs pytest, classifies real vs pollution failures	docs/suite.md
quality	External gates + execution truth. Black, ruff, mypy, CVE scan, coverage	docs/quality.md — Q0-Q8
encoding	Is the source the right encoding? Strict-decodes every text file (UTF-8 by default; configurable)	#encoding + check
integrations	External tools. semgrep, bandit, opt-in MegaLinter, +23 native linters across 17 languages	docs/integrations.md

wiring, phd and runtime aren't Python-only. Python gets the full ast-based rules; every other language gets a portable, dependency-free subset (dead symbols, swallowed errors, dynamic eval, hardcoded secrets, debt markers, unbounded loops) — see Polyglot audits.

Languages

Auto-detects 21 languages through 19 adapters (marker files or source files anywhere in the tree, root included). Python runs the full five-audit stack. Every other language gets a real syntax check, its native test suite, and a wiring/phd/runtime pass — and when a required toolchain is missing, the result is an honest SKIP with the install hint, never a fake pass:

Language	Detection	Syntax check	Test suite	Semantic¹
Python	pyproject.toml, setup.py, *.py	ast.parse per file (built-in)	pytest (via suite audit)	full ast
JS / TS	package.json, *.js, *.ts	node --check; TS via tsc --noEmit (TS1xxx only)	npm test (real script only)	✓
Java	pom.xml, build.gradle, *.java	javac -proc:none (parse errors only; classpath noise not judged)	mvn test / gradlew test	✓
Go	go.mod, *.go	gofmt -l -e (parse + format drift)	go test ./...	✓
Rust	Cargo.toml, *.rs	cargo check	cargo test	✓
C#	*.cs	dotnet build (SKIP if restore fails)	dotnet test	✓
C / C++	CMakeLists.txt, Makefile, *.c(pp)	gcc/clang -fsyntax-only or cl /Zs per unit	ctest (if build/ exists)	phd/runtime¹
Kotlin	*.kt, *.kts	kotlinc	gradlew test	✓
Swift	*.swift	swiftc -typecheck	swift test	✓
PHP	*.php, *.phtml	php -l	vendor/bin/phpunit	✓
Ruby	*.rb	ruby -c	rake	✓
Dart	*.dart	dart analyze	dart test	✓
Scala	*.scala	scalac	sbt test	✓
Elixir	*.ex, *.exs	elixirc	mix test	✓
Zig	*.zig	zig ast-check	zig test	✓
Lua	*.lua	luac -p	—	✓
Haskell	*.hs, *.lhs	stack/cabal dry-run, else ghc -fno-code	stack test / cabal test	phd/runtime¹
HTML / CSS	*.html, *.css, *.scss	tag-balance / brace-balance (structural, stdlib)	—	—
SQL	*.sql	sqlfluff parse (ANSI; SKIP if not installed)	—	phd/runtime¹

¹ Wiring/phd/runtime via the Polyglot audits engine — portable rules for every language, plus a deep tree-sitter AST pass for JS/TS, Rust, Go, Java, C#, Kotlin, Swift, PHP and C/C++. C/C++, Haskell and SQL get phd + runtime but not wiring (linkage/exports make dead-symbol detection unreliable there).

Restrict detection with [audit] languages = ["python", "go"] in audit-code.toml (empty list = auto-detect all).

Polyglot audits

The deep audits go multi-language in two tiers. Python keeps its full 55-rule ast engine; every other language gets a dependency-free, portable rule set (76 phd + 32 runtime regex rules) driven by a per-language LangSpec (file extensions, how a definition looks, which rules apply). Detection is by extension in a single tree walk, so a language needs no toolchain — pure-Kotlin or pure-Elixir repos are scanned out of the box.

On top of that, nine languages get a deep tree-sitter AST pass — 72 structural rules that regex can't reach: useState state that's never read, empty catch blocks found by node type (not brace-guessing), unsafe casts, goto, using namespace in headers, god functions, and more. The grammars ship with the package; if one is missing at runtime, that language's AST pass reports an explicit SKIP note — never a silent downgrade.

20 languages carry rules today:

Tier	Languages
Deep AST (tree-sitter) + portable rules	JS/TS, Rust, Go, Java, C#, Kotlin, Swift, PHP, C/C++
Portable rules (wiring/phd/runtime)	Ruby, Dart, Scala, Elixir, Zig, Lua, Haskell, SQL, CSS/SCSS

What each portable audit looks for:

• wiring — dead symbols: a definition (function/class/method) whose name is never referenced anywhere else. Conservative — exported/pub/capitalised API and entry points are never flagged, so it under-reports rather than cries wolf. (Disabled for C/C++, Haskell and SQL, where linkage/exports make it unreliable.)
• phd — swallowed errors (empty catch), catch-all handlers, dynamic code execution (eval, new Function, Code.eval, load), hardcoded secrets, JS loose equality and React-hooks misuse, Go if err != nil {} and mutex-by-value, Rust .unwrap()/panic! and blocking I/O in async code, Java String ==, C# sync-over-async, Swift try!/fatalError/force-unwrapped optionals, Kotlin !!/GlobalScope, unsafe C (gets/strcpy, unchecked malloc), PHP SQL string interpolation, Haskell unsafePerformIO, SQL DELETE/UPDATE without a WHERE.
• runtime — debug leftovers (console.log, System.out.println, var_dump, IO.inspect), debt markers (TODO/FIXME/XXX/HACK), unbounded loops, SELECT *, CSS !important abuse and z-index escalation.

Findings surface per language as javascript-wiring, go-phd, rust-runtime, kotlin-phd, etc. A language with no rule set returns an honest SKIP. The [audit] languages = [...] allow-list and --wiring/--phd/--runtime module flags apply to these exactly as they do to the Python audits.

Exit codes

Code	Meaning
0	Audits completed, passed
1	Completed but blocking problems found
2	Setup or configuration error
3	Required audit or tool crashed
4	No supported language detected

Severity levels

Every finding has a severity: HIGH, MEDIUM, or INFO. Default reports HIGH only. Use --medium, --info, or --all to expand. The phd audit supports --min-severity=HIGH when run standalone.

The gate

audit-test gate judges only your working-tree diff vs HEAD, inside a disposable git worktree:

• G0 syntax — changed files must parse
• G1 static regression — no new HIGH findings vs HEAD
• G2 suite green — full test suite passes
• G3 execution proof — every changed def + line executes under tests
• G4 mutation kill — injected bugs in changed lines must be caught

Design

• Fail-closed. Crash, missing summary, unparseable file — all failures, never passes.
• Name-level vs execution-level. "Test mentions this" and "body ran" are different facts.
• Judge the diff, not the history. Legacy findings are baseline; only regressions fail.
• Honest limits. No static tool promises semantic correctness — this stack narrows the gap.

Configuration

.audit-test-ignore

Skip directories or files from all scans. Drop this file in your project root. One pattern per line, # for comments. Built-in defaults are always applied (.venv, node_modules, .git, __pycache__, dist, build, etc.).

Use #only blocks to focus audits on specific files:

# .audit-test-ignore
generated/
third_party/
*.pb2.py

#only
fast=[src/main.py,src/cli.py]
slow=[src/quality.py] /mnt/c/other  | full sweep
#only

Group format: name=[file1,file2] [/path_override] [| description]

Patterns match directory/file name parts (exact match, not substring).

Declare the project's expected source encoding with #encoding — used by audit-test check (see Encoding check):

#encoding utf-8

# audit: ok

Add # audit: ok to the end of any line to suppress a finding on that line. Use sparingly — every suppression is counted in the summary.

Applies to the phd, runtime and wiring audits. For wiring, the annotation is honored on the def/class line — or on the closing ): line of a multi-line signature, the natural place to put it.

Only suppress when the audit is wrong: env-var-gated code paths, CLI entry points that can't be covered, and parse-time helpers wiring can't detect through its import-graph walk.

#needs fix

For known issues you can't fix right now but don't want to suppress. Unlike # audit: ok, this does NOT silence the finding — the annotation flags acknowledged debt. Grep the codebase for the current list, or see ROADMAP.md.

Requirements

Tool	Required	Used by
Python 3.10+	✓	all modules
git	✓	gate, suite baseline, wiring (repo root detection)
pytest	✓	suite, quality (Q5 coverage)
coverage	—	quality Q5 (def execution proof)
black	—	quality Q1 / fix mode
ruff	—	quality Q2 / lint / fix mode
mypy	—	quality Q3 (type checking)
pip-audit or safety	—	quality Q4 (CVE scan)
mutmut	—	quality Q8 / gate G4 (mutation testing)
semgrep	—	security integration (structural)
bandit	—	security integration (Python SAST)

Native linters (auto-detected per-language; honest SKIP if tool not installed):

Language	Tools
JS / TS	eslint, prettier
Java	checkstyle, pmd
Go	go vet, golangci-lint
Rust	cargo clippy, rustfmt
C#	dotnet format
C / C++	clang-tidy, cppcheck
HTML / CSS	htmlhint, stylelint
SQL	sqlfluff
PHP	phpstan
Ruby	rubocop
Swift	swiftlint
Kotlin	detekt
Dart	dart analyze
Scala	scalafix
Elixir	credo
Zig	zig fmt
Lua	luacheck
Haskell	hlint

Plus MegaLinter as an opt-in umbrella scan (audit-test megalinter) — thorough but slow, so it never runs by default.

License

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0).

© Yosef Bunick. All rights reserved.

You are free to:

Share — copy and redistribute the material in any medium or format

Under the following terms:

Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. NonCommercial — You may not use the material for commercial purposes. NoDerivatives — If you remix, transform, or build upon the material, you may not distribute the modified material.

License details: https://creativecommons.org/licenses/by-nc-nd/4.0/

This license applies unless otherwise explicitly stated within specific files or directories of this repository.

For permission to monetize, distribute modified versions, remix, sublicense, or commercially use this repository, please contact the creator directly.

AI Training Restriction. No part of this work may be used as training data to train, fine-tune, or otherwise build machine learning models (including large language models, code generation models, or embeddings) without explicit written permission. This restriction applies regardless of whether the output is distributed or kept private. Using this tool to evaluate or benchmark model output is permitted.