fors33-scanner

High-speed file integrity and baseline scanner. Walks one or more roots, measures data gravity (bytes), and classifies large files as attested or unattested based on sibling sidecar presence (.f33, .sig, .asc, .sha256, .sha512, .blake3, .md5, .pem). Emits checksum baselines (Hash Filename format), CSV, or JSON for use with fors33-verifier.

Trust model: The scanner is an O(1) discovery and liability mapping tool based on sidecar presence only. It does not validate Ed25519 signatures or cryptographic proof of baselines. For full cryptographic verification, use fors33-verifier.

For machine parsing, see LLM_CONTEXT.md.

Install

pip install fors33-scanner

Usage

Scan the current directory (default root) with a 1 MB threshold:

fors33-scanner --threshold-mb 1.0

Scan multiple roots:

fors33-scanner --root /var/log --root /data/telemetry --threshold-mb 10

Emit JSON instead of human output (for CI, pipelines):

fors33-scanner --root /data --json

Fail CI/CD when exposure breaches policy threshold:

fors33-scanner --root /data --max-exposure 5.0 --json

Throttle hashing workers for shared runners:

fors33-scanner --root /data --workers 2

Stream SIEM-ready JSONL events (records + summary):

fors33-scanner --root /data --emit-jsonl -

Depth-limit traversal (0=root only, 1=root + direct children):

fors33-scanner --root /data --max-depth 1

Strict audit (fail on permission or file-lock errors instead of skipping):

fors33-scanner --root /data --strict-audit

Single-file scanning:

# Scan a single file
fors33-scanner --root /path/to/file.csv

# Scan a single file with baseline generation
fors33-scanner --root /path/to/file.csv --emit-checksums baseline.txt

# Scan a single file with JSON manifest
fors33-scanner --root /path/to/file.csv --emit-json manifest.json

Single-file mode accepts individual file paths in addition to directories, enabling direct scanning of specific files without directory traversal. By default it recognizes all attestation sidecar extensions (.f33, .sig, .asc, .sha256, .sha512, .blake3, .md5, .pem) for parity with directory scanning.

Pre-0.8.0 stats (below-threshold single-file roots bump skipped_files, and .blake3 siblings are not counted as external attestation):

fors33-scanner --root /data --legacy-scanner-stats

Equivalent: set FORS33_SCANNER_LEGACY_STATS=1.

Record TSA endpoint for tooling that reads FORS33_TSA_URL:

fors33-scanner --tsa-url https://tsa.example.com/rfc3161

Worker count: positive --workers wins; otherwise a positive FORS33_WORKERS; otherwise default_dpk_worker_count() (uses cpu_count and optional FORS33_DPK_MAX_WORKERS). Non-positive values mean auto. Hard cap 64.

Large-file hashing uses FORS33_MMAP_MIN_MB / FORS33_MMAP_MAX_MB (defaults 500 / 4000), clamped to cgroup/RAM ceiling on Linux; optional FORS33_MMAP_PSI_SOME_AVG10_MAX disables mmap under memory pressure.

For production Docker or CI, pin a semver image tag or immutable digest instead of relying on :latest alone.

Generate checksum baseline (sha256, sha512, or blake3 per --algo):

fors33-scanner --root /data --emit-checksums fors33_baseline.sha256
fors33-scanner --root /data --algo sha512 --emit-checksums fors33_baseline.sha512

Emit CSV or JSON baseline (compatible with fors33-verifier):

fors33-scanner --root /data --emit-csv fors33_baseline.csv
fors33-scanner --root /data --emit-json fors33_baseline.json

Add compliance exposure text to human output (default is strictly mathematical):

fors33-scanner --root /data --compliance-report

Exit codes

• 0: successful scan / threshold not breached
• 1: exposure threshold breach (--max-exposure)
• 2: invocation/parameter misuse, or --strict-audit I/O access failure
• 130: user interrupted scan (Ctrl+C)

Output

Default human output (mathematical only):

[FILE COUNT]    : 14,205
[TOTAL BYTES]   : 2.1 TB
[ATTESTED]      : 48 files, 4.1 GB
[UNATTESTED]    : 264 files, 2.1 TB
[ELAPSED]       : 4.20s

Safety and scope

• Read-only: does not modify files or sidecars.
• Scan-only: O(1) discovery; baseline generation uses streaming chunked hashing.
• Excludes common dirs (.git, node_modules, venv, etc). Respects .f33ignore and --ignore-pattern / --exclude-dir.
• Legal notice prints to stderr on startup so data/JSON streams on stdout remain parse-safe.
• See DISCLAIMER.md for enterprise legal/regulatory boundaries.

JSONL contract

• --emit-jsonl PATH emits one flat JSON object per line.
• Multi-root scans include both root_index and root_path in each scan_record.
• timestamp represents hash completion time.
• Final line is scan_summary with aggregate stats and scan parameters.
• If --emit-jsonl - and --json are both requested, JSONL takes precedence on stdout.
• unverified_paths_sample entries (JSON/JSONL consumers): each row includes path, status, and has_sidecar ("true" / "false") since 0.7.1 for integration with seal/re-seal workflows.

Requirements

Python 3.9+. Optional blake3 for BLAKE3 hashing. Linux, macOS, Windows.

License

MIT License. See LICENSE.