certisigma-census 1.5.1

Cryptographic file inventory and exfiltration detection — powered by CertiSigma

CertiSigma Census

Cryptographic file inventory and exfiltration detection — powered by CertiSigma.

Census scans directories, computes SHA-256 hashes, attests them via the CertiSigma API (three-layer cryptographic proof: ECDSA T0, qualified TSA T1, Bitcoin T2), and maintains a local manifest. When suspect files surface, Census compares their hashes against the registry to prove — with cryptographic certainty — whether they match inventoried assets.

Installation

pip install certisigma-census

# With watch mode (filesystem monitoring)
pip install certisigma-census[watch]

# With PDF report generation
pip install certisigma-census[report]

# Everything
pip install "certisigma-census[watch,report]"

Requires Python 3.10+. TOML config support on Python 3.10 uses tomli (auto-installed).

Quick Start

1. Inventory scan

export CERTISIGMA_API_KEY=cs_...

# Scan a directory and attest all file hashes
census scan /path/to/sensitive-files --source inventory-hr

# Dry run — hash only, no attestation
census scan /path/to/files --dry-run

# Scan only PDFs and Word docs, skip files over 100 MB
census scan /data --include "*.pdf" --include "*.docx" --max-size 100M

# Resume an interrupted scan
census scan /data --source quarterly --manifest inventory.db --resume

# Parallel hashing for large directories (4 CPU cores)
census scan /data --workers 4

# Attest the manifest itself (proves manifest existed at scan time)
census scan /data --attest-manifest

This produces a .census-manifest.db (SQLite) mapping each hash to its file path, size, and attestation metadata.

2. Breach comparison

# Compare suspect files against the CertiSigma registry
census compare /path/to/suspect-files --manifest /path/to/.census-manifest.db

# Save report as JSON or CSV
census compare /suspect --output report.json
census compare /suspect --output report.csv

Exit code: 0 if no matches, 1 if matches found.

3. Manifest status and export

# Show summary
census status /path/to/.census-manifest.db

# Export manifest as CSV for compliance reporting
census export manifest.db --format csv --output inventory.csv

# Export as JSON
census export manifest.db --format json --output inventory.json

# Export as sha256sum (GNU coreutils compatible — works with sha256sum -c)
census export manifest.db --format sha256sum --output checksums.sha256

4. Evidence verification

# Verify a hash against the CertiSigma registry
census verify a1b2c3d4e5f67890...

# Verify a file (hash it first, then check)
census verify /path/to/document.pdf --file

# Full-chain manifest verification (all hashes against the registry)
census verify-manifest inventory.db --strict
census verify-manifest inventory.db --detailed --json

# Hash from stdin (for pipes and CI/CD)
echo "data" | census hash --stdin

# Save OpenTimestamps proof
census verify a1b2c3... --save-ots proof.ots

No API key required — all verification endpoints are public.

5. Integrity check

# Check files against manifest baseline
census integrity manifest.db

# Strict mode: exit 1 on any discrepancy
census integrity manifest.db --strict

100% local operation — no API calls, no network needed.

5b. Update baseline (AIDE-style)

# Accept verified changes into manifest (interactive confirmation)
census update manifest.db

# Non-interactive (CI/cron)
census update manifest.db --yes

# Then attest new hashes
census scan /data --resume --manifest manifest.db

Completes the FIM workflow: detect → review → accept. New entries are unattested until the next scan.

6. Forensic reports

# HTML report (always available, zero dependencies)
census report manifest.db -o report.html

# PDF report (requires: pip install certisigma-census[report])
census report manifest.db -o report.pdf --evidence --integrity

# Evidence bundle: ZIP with report + OTS proofs + checksums
census report manifest.db -o bundle.zip --bundle --evidence

7. Manifest diff

# Compare two manifests
census diff baseline.db current.db

# HTML diff report
census diff baseline.db current.db -o diff.html

# Machine-readable (exit codes: 0=none, 1=added, 2=removed, 4=modified)
census diff baseline.db current.db --json

8. Standalone hashing

# Hash a file
census hash document.pdf

# Hash a directory
census hash /path/to/files

# Verify against known hash
census hash document.pdf --verify a1b2c3d4e5...

9. Attestation tracking

# Check attestation status
census track att_12345

# Wait for Bitcoin anchoring
census track att_12345 --poll --timeout 7200

10. Self-diagnostic

# Run all health checks
census doctor

# Check including a specific manifest
census doctor --manifest inventory.db

# Machine-readable output for CI
census doctor --json

11. Manifest merging

# Merge manifests from different servers
census merge server1.db server2.db -o combined.db

# Merge with glob
census merge scans/*.db -o full-inventory.db --json

12. Audit log

# View all operations
census audit-log show

# Verify hash chain integrity
census audit-log verify

# Machine-readable
census audit-log show --last 10 --json

13. Named snapshots

# Create a compliance baseline
census snapshot create q1-baseline --manifest inventory.db

# List snapshots
census snapshot list

# Compare two snapshots
census snapshot diff q1-baseline q2-baseline

14. Forensic annotation

# Annotate an attestation with case metadata
census annotate att_123 --note "Evidence for case FR-2026-42" --tag "case-2026-001"

# Zero-knowledge mode: encrypt before sending
census annotate att_123 --note "Confidential" --encrypt --encryption-key <key>

# GDPR right-to-erasure
census annotate att_123 --delete

15. Configuration

# Create config template
census config init --project

# View effective config
census config show

# Enable shell completions
eval "$(census completion bash)"

16. Forensic share tokens

# Create a share token (chain of custody)
census share create <att_id> --expires 24h --recipient "Legal Dept" --max-uses 5

# List / inspect / revoke
census share list --json
census share info <token_id>
census share revoke <token_id>

17. Structured tagging

# Tag attestations for classification
census tag set <att_id> -t department=legal -t case=2026-001

# Encrypted tags (zero-knowledge)
census tag set <att_id> -t classification=confidential --encrypt

# Query by tags (AND logic, cursor pagination)
census tag query -f department=legal --limit 50 --json

18. Key rotation

# Rotate encryption key (NIST SP 800-57)
census key-rotate <att_id> --old-key <hex64> --new-key <hex64>

19. Derived lists (third-party breach detection)

# Create an opaque HMAC-SHA256 derived list from your manifest
census derived-list create --manifest ./manifest.db --label "Q1 2026"

# Third party matches their suspects (server never sees plaintext)
census derived-list match <list_id> --list-key <hex64> --hashes-file suspects.txt

# Audit trail
census derived-list access-log <list_id>

20. Metadata read

census metadata get <att_id> --json
census metadata get <att_id> --decrypt --encryption-key <hex64>

21. Watch mode (continuous monitoring)

# Watch a directory for changes and attest new/modified files
census watch /path/to/files --source "production"

# Dry run — hash only, no attestation
census watch /data --dry-run

# Network mount — use polling
census watch /mnt/share --polling --poll-interval 10

Requires: pip install certisigma-census[watch]

22. Manifest seal (tamper evidence)

# Create an HMAC-SHA256 seal for a manifest
census seal ./manifest.db --key $(census key-gen)

# Verify the seal before trusting a manifest
census verify-seal ./manifest.db --key <hex64>

# JSON output
census verify-seal ./manifest.db --key <hex64> --json

The seal proves the manifest has not been modified since it was sealed. Follows the Tripwire/AIDE signed-database pattern.

23. Quiet mode (scripting)

# Suppress informational output — only errors and exit codes
census -q scan /data --dry-run
census -q compare /suspects

# Quiet + JSON — clean machine-readable output
census -q scan /data --json --attest-manifest

24. Bulk leak detection

# Scan a suspect drive against your org inventory (up to 50K hashes/call)
census bulk-scan /mnt/suspect-drive --json

# Cross-reference with a local manifest
census bulk-scan ./data --manifest inventory.db --workers 4

# Dry run — hash and count, no API call (save rate limit)
census bulk-scan /data --dry-run

# Label the scan for incident tracking
census bulk-scan /exports --source incident-2026-003 --json

# Save results to file
census bulk-scan /exports --output results.json

# Report-only mode — always exit 0 (for CI pipelines)
census bulk-scan /data --exit-zero --json > results.json

# Summary mode — counts only, no match details
census bulk-scan /data --summary --exit-zero

Exit code: 0 if no matches (or --exit-zero), 1 if matches found (potential exfiltration).

25. Organization statistics

# View org-level inventory stats
census stats

# Machine-readable
census stats --json

26. SARIF output (CI/CD integration)

# Compare with SARIF output for GitHub Security tab
census compare /suspects --format sarif > results.sarif

# Write SARIF directly to file (recommended for CI/CD)
census compare /suspects --format sarif --output results.sarif

# Report-only mode — always exit 0 (upload SARIF without pipeline failure)
census compare /suspects --format sarif --output results.sarif --exit-zero

# Summary mode — counts only, concise CI logs
census compare /suspects --summary --exit-zero

# SARIF + JSON are also available
census compare /suspects --format json

SARIF v2.1.0 output can be uploaded to GitHub Security tab, VS Code SARIF Viewer, and other compatible tools.

27. JSONL streaming output

# Stream results to a log file (one JSON object per line)
census compare /suspects --format jsonl >> /var/log/census/matches.jsonl

# Pipe to jq for real-time filtering
census compare /suspects --format jsonl | jq 'select(.level=="T2")'

# JSONL is available on compare, bulk-scan, integrity, verify-manifest, and diff
census integrity manifest.db --format jsonl
census diff base.db current.db --format jsonl

28. On-match notification hooks

# Execute a command when matches are found (JSON on stdin)
census compare /suspects --on-match './scripts/alert.sh'

# POST to a webhook
census compare /suspects --on-match 'curl -s -X POST -d @- https://hooks.slack.com/...'

# Also available on bulk-scan
census bulk-scan /data --on-match 'python3 scripts/notify.py'

The --on-match command is only executed when matches > 0. Match data (JSON) is piped to stdin.

How It Works

1. Scan — Census walks the directory, computes SHA-256 for each file (streamed, constant memory), and builds a local manifest.
2. Attest — Hashes are sent in batches (up to 100 per call) to the CertiSigma API. Each hash receives a three-layer cryptographic proof (T0 ECDSA signature, T1 qualified TSA timestamp, T2 Bitcoin anchor).
3. Compare — Suspect files are hashed and verified against the registry via POST /verify/batch. Matches prove the file was previously inventoried, regardless of filename or directory structure changes.

The original file content never leaves the client. Only SHA-256 hashes are transmitted.

Features

Feature	Description	Docs
File filters	--include, --exclude globs; --min-size, --max-size	scanning.md
Resume scans	--resume skips unchanged files, preserves attestation state	scanning.md
CSV/JSON export	Compare reports and manifest export in both formats	comparison.md
Retry with backoff	Automatic retry on 429/5xx with exponential backoff	retry-and-resilience.md
Structured logging	--log-format json for SIEM/ELK integration	logging.md
Progress bars	Visual feedback for scan, attest, and compare operations	scanning.md
SQLite manifest	WAL mode, indexed lookups, auto-migration from JSON	manifest.md
Watch mode	Continuous filesystem monitoring with batch attestation	watching.md
Evidence verification	Full T0/T1/T2 chain, OTS proof export	evidence.md
Integrity check	Tamper detection against manifest baseline	integrity.md
Forensic reports	HTML, PDF, evidence bundles (ZIP)	reporting.md
Manifest diff	Compare snapshots, AIDE-style exit codes, HTML reports	diff.md
Standalone hashing	SHA-256 without manifests or API calls	hash.md
Attestation tracking	Monitor T0/T1/T2 progression with --poll	tracking.md
Config files	TOML config with user/project precedence	config.md
Shell completions	bash, zsh, fish via census completion	—
Self-diagnostic	API health, config, inotify, manifest integrity	doctor.md
Manifest merging	Combine manifests from distributed scans	merge.md
JSON output	--json on scan, compare, status, doctor, merge	—
Audit log	Tamper-evident JSONL with SHA-256 hash chain	audit-log.md
Named snapshots	Compliance baselines with diff comparison	snapshots.md
Forensic annotation	Metadata, tags, case IDs on attestations	annotate.md
Zero-knowledge encryption	AES-256-GCM client-side metadata encryption	annotate.md
Forensic sharing	Time-limited, use-limited share tokens (chain of custody)	sharing.md
Structured tagging	Key-value classification with encrypted tags and query	tagging.md
Key rotation	NIST SP 800-57 AES-256 key rotation for metadata + tags	key-rotation.md
Derived lists	HMAC-SHA256 opaque third-party breach detection	derived-lists.md
Metadata read	Read attestation metadata with optional decryption	—
Manifest seal	HMAC-SHA256 tamper-evidence seal (Tripwire/AIDE pattern)	seal.md
Quiet mode	--quiet / -q suppresses info output for scripting	—
Manifest self-attestation	--attest-manifest anchors manifest hash at scan time	—
Bulk leak detection	bulk-scan — 50K hashes/call, --dry-run, --source, --output	—
Organization stats	stats — total claims, unique hashes, monthly breakdown	—
SARIF output	compare --format sarif — v2.1.0 with help, tags, invocations, file write	—
Baseline update	update — AIDE-style accept verified changes into manifest (detect → review → accept)	—
JSONL streaming	--format jsonl on compare, bulk-scan, integrity, verify-manifest, diff	—
On-match hooks	--on-match CMD — execute command with results on stdin (compare, bulk-scan)	—
CI/CD integration	--exit-zero (report-only mode), --summary (counts only) on compare and bulk-scan	—
--no-color	Disable colored output; also respects NO_COLOR env var (no-color.org)	—
Forensic JSON metadata	census_version and elapsed_seconds in all JSON output	—

Full documentation: docs/features/

CLI Reference

Global options

Option	Description
-v / --verbose	Enable debug logging
-q / --quiet	Suppress informational output (errors and --json always shown)
--log-format text|json	Log output format (default: text)
--no-color	Disable colored output (also respects NO_COLOR env, see no-color.org)
--version	Show version

census scan

Option	Description
--source LABEL	Source label for attestations
--manifest PATH	Manifest output path (default: <dir>/.census-manifest.db)
--api-key KEY	API key (or set CERTISIGMA_API_KEY)
--base-url URL	Override API base URL
--dry-run	Hash only, no attestation
--resume	Resume interrupted scan
--include GLOB	Include files matching pattern (repeatable)
--exclude GLOB	Exclude files matching pattern (repeatable)
--min-size SIZE	Skip files smaller than SIZE (e.g. 1K, 10M)
--max-size SIZE	Skip files larger than SIZE (default: 5G)
--workers N	Parallel hashing workers (default: 1, max: 8)
--attest-manifest	Attest the manifest's own SHA-256 after scan
--json	Machine-readable JSON summary

census compare

Option	Description
--manifest PATH	Local manifest for cross-referencing
--output PATH	Save report (.json or .csv by extension)
--format text|json|sarif|jsonl	Output format (default: text). sarif emits SARIF v2.1.0; jsonl streams one JSON object per match
--include/--exclude/--min-size/--max-size	Same filters as scan
--detailed	Enriched results: source label, T0/T1/T2 level (requires API key)
--workers N	Parallel hashing workers (default: 1, max: 8)
--json	Machine-readable JSON output (equivalent to --format json)
--exit-zero	Always exit 0 (report-only mode for CI pipelines)
--summary	Show only counts, no match details
--on-match CMD	Execute CMD with match results as JSON on stdin (only if matches > 0)

census export

Option	Description
--format csv|json|sha256sum	Output format (default: csv)
--output PATH	Output file (default: stdout)

census verify

Option	Description
--file	Treat argument as a file path (hash it first)
--save-ots PATH	Save OTS proof to this path
--json	Machine-readable JSON output
--api-key KEY	API key (optional for verify)
--base-url URL	Override API base URL

census verify-manifest

Option	Description
--detailed	Fetch enriched data (source, level) per hash
--strict	Exit with code 1 if any hash is not attested
--json	Machine-readable JSON output
-o/--output PATH	Save report (.csv or .json)
--api-key KEY	API key (optional, needed for --detailed)
--base-url URL	Override API base URL

census integrity

Option	Description
--json	Machine-readable JSON output
--format text|json|jsonl	Output format (default: text)
--output PATH	Save results (.csv or .json by extension)
--strict	Exit with code 1 on any discrepancy

census update

Option	Description
--yes / -y	Skip confirmation prompt (non-interactive)
--json	Machine-readable JSON output

Runs integrity check, then applies changes (remove missing, re-hash modified, add new). New entries are attested=False.

census report

Option	Description
-o/--output PATH	Output file (.html, .pdf, or .zip) required
--evidence	Fetch T0/T1/T2 evidence chain for attested files
--integrity	Run integrity check and include results
--bundle	Generate evidence bundle (ZIP)
--api-key KEY	API key (needed only with --evidence)

census status

Option	Description
--json	Machine-readable JSON output

census doctor

Option	Description
--manifest PATH	Check health of a specific manifest file
--json	Machine-readable JSON output
--api-key KEY	API key
--base-url URL	Override API base URL

census merge

Option	Description
-o/--output PATH	Output manifest path required
--json	Machine-readable JSON summary

census diff

Option	Description
--json	Machine-readable JSON output
-o/--output PATH	Save report (.html, .csv, or .json by extension)
--summary	Show only counts, no individual file details

Exit codes: 0=none, 1=added, 2=removed, 4=modified (bitmask, OR'd together).

census hash

Option	Description
--stdin	Read data from stdin instead of a file
--verify HASH	Compare computed hash against expected SHA-256
--json	Output as JSON array

census track

Option	Description
--poll	Continuously check until T2 level reached
--poll-interval SECS	Seconds between checks (default: 60)
--timeout SECS	Max time to poll (default: 3600)
--json	Machine-readable JSON output
--api-key KEY	API key
--base-url URL	Override API base URL

census config

Action	Description
show	Display effective merged config
init	Create a template config file
paths	Show config file locations
--project	Act on project .census.toml

census audit-log

Action	Description
show	Display audit log entries
verify	Check hash chain integrity
clear	Delete the audit log file
--log-path PATH	Override audit log file path
--last N	Show only last N entries (with show)
--json	Machine-readable JSON output

census snapshot

Action	Description
create <name>	Save a named snapshot of a manifest
list	List all snapshots
diff <name1> <name2>	Compare two snapshots
delete <name>	Remove a snapshot
--manifest PATH	Manifest to snapshot (required for create)
--snapshot-dir PATH	Override snapshot directory
--json	Machine-readable JSON output

census annotate

Option	Description
--note TEXT	Free-text note
--tag TEXT	Tag label (e.g. case number)
--case-id TEXT	Forensic case identifier
--source TEXT	Update source label
--delete	Soft-delete metadata (GDPR)
--encrypt	Encrypt client-side (AES-256-GCM)
--encryption-key HEX	64-char hex AES-256 key
--decrypt	Decrypt and display stored metadata
--json	Machine-readable JSON output
--api-key KEY	API key

census share

Action / Option	Description
create <att_id>...	Create share token for attestation(s)
list	List all share tokens
info <token_id>	Inspect a specific token
revoke <token_id>	Revoke a token
--expires DURATION	Token lifetime: 30m, 24h, 7d (default: 24h)
--recipient TEXT	Recipient label
--max-uses N	Max usage count
--json	Machine-readable JSON output

census tag

Action / Option	Description
set <att_id>	Set tags (requires -t key=value)
get <att_id>	List tags on an attestation
delete <att_id> <key>	Delete a specific tag
query	Query attestations by tag filter
-t, --tag key=value	Tag pair (repeatable)
-f, --filter key=value	Query filter (repeatable, AND logic)
--encrypt	Encrypt tag values (AES-256-GCM)
--decrypt	Decrypt on get
--limit N	Max query results (default: 100)
--cursor TOKEN	Pagination cursor
--json	Machine-readable JSON output

census key-rotate

Option	Description
<attestation_id>	Target attestation
--old-key HEX	Current 64-char hex AES-256 key
--new-key HEX	New 64-char hex AES-256 key
--json	Machine-readable JSON output

census derived-list

Action / Option	Description
create	Create HMAC-SHA256 derived list
list	List all derived lists
info <list_id>	Get list details
match <list_id>	Match suspect hashes against list
access-log <list_id>	View access audit trail
signature <list_id>	ECDSA signature verification (no auth required)
revoke <list_id>	Revoke a list
--manifest PATH	Manifest to read hashes from
--tag-filter JSON	JSON tag filter for server-side selection
--label TEXT	Human-readable label
--expires HOURS	Expiry in hours (max 2160)
--list-key HEX	HMAC key (64 hex chars) for match
--hashes-file PATH	File with one hash per line for match
--json	Machine-readable JSON output

census metadata

Action / Option	Description
get <att_id>	Read attestation metadata
--decrypt	Decrypt encrypted extra_data
--encryption-key HEX	64-char hex AES-256 key
--json	Machine-readable JSON output

census key-gen

Generate a random AES-256 encryption key (64 hex characters, 256 bits). The key is shown only once — store it securely.

census key-gen              # outputs the key to stdout
census key-gen --json       # JSON output: {"key": "...", "algorithm": "AES-256-GCM", "bits": 256}

census completion

Takes a shell name: bash, zsh, or fish.

eval "$(census completion bash)"   # bash
eval "$(census completion zsh)"    # zsh
census completion fish | source    # fish

census watch

Option	Description
--debounce SECS	Quiet period before processing (default: 2.0s)
--batch-interval SECS	Max time between attestation batches (default: 30s)
--scan-on-start / --no-scan-on-start	Baseline scan before watching (default: on)
--on-delete ignore|mark|remove	Action on file deletion (default: ignore)
--polling	Use PollingObserver for NFS/CIFS mounts
--poll-interval SECS	Polling interval (default: 5s)
--source/--manifest/--api-key/--dry-run	Same as census scan
--include/--exclude/--min-size/--max-size	Same filters as scan

Requires: pip install certisigma-census[watch]

census seal

Option	Description
MANIFEST_PATH	Path to the manifest file
--key KEY	HMAC key (64 hex chars = 256 bits)
--json	Machine-readable JSON output

census verify-seal

Option	Description
MANIFEST_PATH	Path to the manifest file
--key KEY	HMAC key used to create the seal
--json	Machine-readable JSON output

Exit code 0 = valid, 1 = invalid or error.

census bulk-scan

Option	Description
SUSPECT_DIR	Directory to scan
--manifest PATH	Local manifest for cross-referencing original paths
--include/--exclude/--min-size/--max-size	Same filters as scan
--workers N	Parallel hashing workers (default: 1, max: 8)
--source LABEL	Source label for audit logging (e.g. incident ID)
--dry-run	Hash only, no API call — preview file/hash/chunk counts
--output PATH	Save results to JSON file
--json	Machine-readable JSON output
--exit-zero	Always exit 0 (report-only mode for CI pipelines)
--summary	Show only counts, no match details
--api-key KEY	API key (requires scan scope)
--base-url URL	Override API base URL

Uses POST /scan — up to 50K hashes per call with automatic chunking. Exit code: 0=no matches, 1=matches found (or always 0 with --exit-zero).

census stats

Option	Description
--json	Machine-readable JSON output
--api-key KEY	API key (requires batch scope)
--base-url URL	Override API base URL

Dependencies

• certisigma — Official CertiSigma Python SDK
• click — CLI framework

Optional:

• watchdog — Filesystem monitoring (only for census watch)
• fpdf2 — PDF report generation (only for census report with .pdf output)

License

MIT — Ten Sigma Sagl