DocGuard — Multi-Format Document Security Scanner

DocGuard is a Python security pipeline that statically analyzes documents submitted by external suppliers and automatically routes them based on risk. Suspicious files are quarantined; clean files are forwarded with macros intact for downstream use.

It is packaged as a pip-installable library so it can be imported directly into existing company projects.

---

Content-Based Routing

As of this version, DocGuard determines a file's format from its actual content, not its filename extension. Previously, a malicious file renamed from payload.exe to invoice.pdf would be routed to PDFScanner purely because of the .pdf name — the file's real content was never verified.

FileRouter now sniffs the file (via docguard/content_sniffer.py) using:

• Magic bytes for binary formats — PDF (%PDF-), PNG, JPEG, LNK, ZIP, OLE2
• ZIP-internal inspection for OOXML — reads [Content_Types].xml to distinguish .docx/.xlsx/.docm/.xlsm/.dotm/.xltm/.xlsb, since they all share the same outer ZIP signature
• OLE2-internal inspection for legacy Office — reads internal stream names (WordDocument, Workbook/Book, MAPI property streams) to distinguish .doc/.xls/.msg, since they all share the same outer OLE2 signature
• Structural heuristics for text formats without magic bytes — .xml (<?xml), .html (<!doctype html), .url ([InternetShortcut]), .eml (RFC822 headers)

No new heavy dependency (no python-magic/libmagic) was introduced — olefile is already a transitive dependency of oletools, so detection stays dependency-light and Windows-install-friendly.

When content and extension disagree: the file is still scanned and routed according to its real detected content (not silently trusted based on the filename), and an Extension_Content_Mismatch finding (weight 50) is added to the top of the findings list — so the disguise itself becomes part of the audit trail and risk score, on top of whatever the real scanner finds.

The one genuine limitation: CSV vs Markdown

.csv and .md are the two formats with no reliable content signature at all — plain delimited text and plain prose share no structural markers a sniffer can key on. classify_ambiguous_text() uses lightweight heuristics (consistent delimiter count across rows vs. Markdown syntax like #, ```, [text](url)) and falls back to the claimed extension when the signal is genuinely inconclusive, rather than guessing. This is an intentional, documented gap — not an oversight.

If content cannot be identified as anything

A file whose content matches no signature at all, and isn't a plausible ambiguous-text fallback, is rejected outright with a clear error — DocGuard no longer silently scans unidentifiable content just because its extension looked legitimate.

---

Supported File Formats

Format	Extensions	Scanner
Excel (macro-enabled)	.xlsm .xls .xlsb .xltm	VBA + YARA + XLM
Word (macro-enabled)	.doc .docm .dotm	VBA + YARA
PDF	.pdf	JS/action detection
Images	.jpg .jpeg .png	Binary + EXIF + decompression bomb + OCR text scan
Office Open XML	.docx .xlsx	DDE + relationship + HTML-injection scan
Outlook / MIME email	.msg .eml	Phishing + attachment + oversized-file scan
Markdown / text	.md	Link + script injection scan
CSV data	.csv	Formula-injection scan
XML data	.xml	XXE + malformed-XML scan
HTML	.html	Dangerous markup + event-handler scan
Windows shortcut	.lnk	Magic-byte + dangerous-target scan
Internet shortcut	.url	Dangerous URL-scheme scan

---

Installation

Requirements: Python 3.12

# Clone the repository
git clone https://github.com/niyabraham/DocGuard.git
cd DocGuard

# Install the package
py -3.12 -m pip install -e .

All Python dependencies (oletools, yara-python, pypdf, Pillow, openpyxl, extract-msg, XLMMacroDeobfuscator, pytesseract) are installed automatically.

Optional system dependencies

Two detection layers shell out to system binaries that are not installed by pip, and degrade to a zero-weight informational finding (rather than failing) if missing:

Binary	Enables	Install
clamscan (ClamAV)	Known-malware signature scan — cross-cutting, runs against every file regardless of format (see docguard/clamav_scanner.py)	apt install clamav (Linux) — then run freshclam to download the virus database. Without a database loaded, scans report ClamAV_NoDatabase rather than silently skipping.
tesseract (Tesseract OCR)	OCR text scan on images (Layer 3 of the image scanner) — catches phishing lures or "enable macros" prompts rendered as picture text specifically to dodge text-based scanners	apt install tesseract-ocr (Linux)

Check docguard/clamav_scanner.CLAMSCAN_AVAILABLE and docguard/scanners/image_scanner.OCR_AVAILABLE to see whether either was detected at import time.

---

CLI Usage

Scan a file

docguard sample_files\invoice.xlsm --quarantine-dir quarantine

Output:

[*] Initializing DocGuard Pipeline for: sample_files\invoice.xlsm
[+] Scan Complete.
    - Format         : Excel Workbook (Macro-enabled)
    - Verdict        : BLOCKED
    - Risk Score     : 400 / threshold 50
    - Destination    : quarantine\invoice_quarantined_20260811120000.xlsm
    - Findings Count : 13

Files scoring at or above the per-format threshold are BLOCKED and copied into --quarantine-dir (if given — omit it for a results-only outcome, see Results-only mode below). Files scoring below the threshold are CLEAN and copied into --clean-output-dir with macros intact (if given — same rule, no default location).

Triage quarantined files

docguard --triage

Lists every quarantined file with its SHA-256 hash, file size, risk score, findings count, and scan timestamp — ready for analyst review.

---

Python API Usage

Import DocGuard directly into an existing project:

from docguard import DocGuardPipeline

result = DocGuardPipeline("path/to/supplier_file.xlsm").execute()

print(result["verdict"])        # "CLEAN" or "BLOCKED"
print(result["risk_score"])     # e.g. 400
print(result["risk_threshold"]) # e.g. 50
print(result["format"])         # e.g. "Excel Workbook (Macro-enabled)"
print(result["file_copied"])    # True/False — whether a physical copy was made
print(result["destination"])    # path where file was copied, or None

for finding in result["findings"]:
    print(finding["rule"], finding["weight"], finding["desc"])

Results-only mode

Neither quarantine_dir nor clean_output_dir has a default location — DocGuardPipeline only copies a file if the matching path was actually given, for either verdict. This copy step is optional — the scan result (verdict, risk score, findings, audit log entry) is the actual deliverable of a scan, and a host application that already manages its own file storage shouldn't be forced to accept a duplicate copy of every file it submits, BLOCKED or CLEAN.

# Results-only: nothing is copied anywhere, for either verdict
result = DocGuardPipeline("supplier_file.xlsm").execute()

# verdict, risk_score, and findings are fully computed exactly as before
# result["file_copied"] is False, result["destination"] is None
# the scan is STILL logged in full to logs/audit_log.jsonl — that part
# is unconditional, since storing the result is the core objective

# Copy a BLOCKED file into one directory, a CLEAN file into another —
# independent of each other, each opt-in on its own terms
result = DocGuardPipeline(
    "supplier_file.xlsm",
    quarantine_dir="D:/quarantine",
    clean_output_dir="D:/clean_output",
).execute()

Equivalent CLI flags:

docguard supplier_file.xlsm                                                          # results-only, nothing copied
docguard supplier_file.xlsm --quarantine-dir D:\quarantine                            # BLOCKED files copied here
docguard supplier_file.xlsm --clean-output-dir D:\clean_output                        # CLEAN files copied here
docguard supplier_file.xlsm --quarantine-dir D:\quarantine --clean-output-dir D:\clean # both, independently

--triage only reflects scans that were actually copied into a quarantine directory, since it lists files that physically exist there — pass the same --quarantine-dir you scanned with.

Other importable classes

from docguard import FileRouter        # route a file to its scanner
from docguard import QuarantineManager # manage quarantine directory
from docguard import MacroDeobfuscator # deobfuscate VBA snippets directly
from docguard import FORMAT_ROUTER     # dict of supported extensions
from docguard import detect_format     # sniff a file's real content type

---

Analysis Pipeline

Every file passes through the same top-level pipeline; the per-format scanner step itself is broken into layers specific to that format (the macro scanner's six layers are detailed below as an example):

DocGuardPipeline.run()
│
├── FileRouter            — selects scanner and threshold by detected content type
│
├── Scanner.analyze()          — format-specific (e.g. macro scanner's 6 layers, below)
│
├── ClamAV signature scan      — cross-cutting: runs against EVERY file,
│                                 regardless of detected format (docguard/clamav_scanner.py)
│
├── Second-pass deobfuscation (base64 feedback into risk score)
│
├── Verdict & routing
│   ├── score >= threshold → BLOCKED → quarantine_dir, if one was given
│   └── score <  threshold → CLEAN  → clean_output_dir, if one was given
│
└── Audit log → logs/audit_log.jsonl

Layer 2 runs before Layers 4–6. This is the critical architectural decision: deobfuscated strings (e.g. Chr() sequences reconstructed into "Shell") are visible to all downstream rule matching. Without this ordering, obfuscated macros evade detection entirely.

---

Detection Layers in Detail

Layer 1 — VBA Extraction (olevba)

Extracts all VBA macro code streams and scores keywords by type:

• AutoExec triggers (Auto_Open, Workbook_Open, Document_Open): weight 30
• Dangerous API calls (Shell, CreateObject, Environ): weight 30
• IOC keywords (cmd.exe, payload.exe): weight 30
• General suspicious keywords: weight 10

Layer 2 — Deobfuscation

Reconstructs hidden strings before any rule matching:

• Chr() decoding: Chr(83) & Chr(104) & Chr(101) & Chr(108) & Chr(108) → "Shell"
• String concatenation collapse: "She" & "ll" → "Shell"
• Base64 extraction: detects and decodes embedded base64 payloads

Layer 3 — XLM / Excel 4.0 Macro Detection

Detects legacy Excel 4.0 macro sheets (XLM), which are a separate execution path from VBA and commonly used to evade VBA-only scanners.

Layer 4 — Custom Rule Matching

Weighted substring rules defined in rules/macro_rules.json. Externalized so new rules can be added without touching Python code.

Layer 5 — YARA Signatures

33 YARA rules across 6 files covering:

File	Coverage
vba_autoexec.yar	Auto-execution entry points
vba_shell_execution.yar	Shell, PowerShell, CMD, COM object abuse
vba_download_cradles.yar	HTTP download, WebClient, URLDownloadToFile
vba_obfuscation.yar	Chr(), base64, StrReverse, hex construction
vba_persistence.yar	Registry Run keys, startup folder, scheduled tasks
vba_process_injection.yar	VirtualAlloc, WriteProcessMemory, NT APIs

Each YARA rule carries its own weight in rule metadata — no weights are hardcoded in Python.

Layer 6 — Keyword Co-occurrence Heuristic

Fires when 2 or more behavioural trigger keywords appear together in the same document (shell, createobject, wscript.shell, environ, exec, powershell). Co-occurrence signals chained behaviour rather than isolated keyword presence: weight 40.

---

Cross-Cutting Layers

Two layers sit outside the per-format scanners above — they don't belong to any one file type, so they run differently:

ClamAV Signature Scan

Runs against every file, regardless of detected format — wired directly into DocGuardPipeline.run() rather than into any single scanner class (see docguard/clamav_scanner.py). Shells out to clamscan (no clamd daemon required) and matches known-malware signatures from the local virus database. A signature match is weighted heavily (100) since it's near-certain confirmation, not a heuristic. Degrades gracefully — and visibly, via a zero-weight finding — if clamscan isn't installed or has no database loaded (run freshclam to fetch one). See Optional system dependencies.

Image OCR Text Scan (Image Scanner, Layer 3)

Extracts visible text rendered into an image using Tesseract, then runs it through the same script-pattern and suspicious-URL checks as Layers 1–2, plus a dedicated phishing-phrase check ("verify your account", "enable macros", "account has been suspended", etc). This catches content a byte-level or metadata scan can't see at all: a phishing lure or a fake "enable macros to continue" prompt rendered as a picture specifically to dodge scanners that only look at machine-readable text. Weight 35 for script/shell content, 20 per phishing phrase — multiple phrases stack, so a convincing lure image reliably clears the image threshold (65) on its own.

---

Per-Format Risk Thresholds

Thresholds vary by format because threat severity and false-positive rates differ:

Format	Threshold	Rationale
Macro-enabled Office	50	Highest risk — direct code execution
PDF	60	Higher FP rate — many legitimate PDFs use JS/forms
OOXML (docx/xlsx)	55	DDE and external links — moderate risk
Outlook email	55	Phishing + attachment risk
Images	65	Rare threat vector — polyglot/EXIF abuse
Markdown	70	Lowest risk — requires most evidence to block

---

Output Directories

Both are optional and only created when actually needed — neither has a default. Pass --quarantine-dir PATH to have a BLOCKED file copied there, and/or --clean-output-dir PATH to have a CLEAN file copied there. Omit either flag and that verdict's file is scanned and logged, not copied. Either directory is created automatically in the caller's working directory — not inside the installed package — the first time a file actually needs to land in it, never just because a path was configured.

<quarantine_dir>    BLOCKED files copied here (original preserved at source), only if given
<clean_output_dir>  CLEAN files copied here with macros intact, only if given
logs/               audit_log.jsonl — append-only scan records, ALWAYS written

logs/ is the one directory that's always created and always written to, regardless of quarantine_dir/clean_output_dir — storing the scan result is DocGuard's core objective; copying the file itself is a convenience on top of that, opt-in per verdict.

Audit Log Format

Every scan appends one JSON line to logs/audit_log.jsonl:

{
    "timestamp": "2026-08-11 23:57:32",
    "filename": "chr_obfuscated_macro.xlsm",
    "file_path": "C:\\...\\sample_files\\chr_obfuscated_macro.xlsm",
    "format": "Excel Workbook (Macro-enabled)",
    "verdict": "BLOCKED",
    "risk_score": 400,
    "risk_threshold": 50,
    "destination": "C:\\...\\quarantine\\chr_obfuscated_macro_quarantined_20260811235732.xlsm",
    "findings_count": 13,
    "findings": [...]
}

---

Project Structure

docguard/
├── pyproject.toml                  pip packaging configuration
├── generate_test_samples.py        generates test files (dev only)
│
├── docguard/                     installable package
│   ├── __init__.py                 public API (DocGuardPipeline, FileRouter, ...)
│   ├── __main__.py                 enables python -m docguard
│   ├── cli.py                      docguard console script entry point
│   ├── pipeline.py                 end-to-end orchestration + audit logging
│   ├── file_router.py              format detection and scanner dispatch
│   ├── content_sniffer.py          content-based format detection
│   ├── deobfuscator.py             Chr(), concat, base64 reconstruction
│   ├── quarantine_manager.py       file isolation + triage reporting
│   ├── clamav_scanner.py           cross-cutting ClamAV signature scan (all formats)
│   │
│   ├── scanners/
│   │   ├── macro_scanner.py        VBA + XLM + YARA (Excel/Word macro files)
│   │   ├── pdf_scanner.py          PDF JS/action/URL analysis
│   │   ├── office_scanner.py       OOXML DDE + relationship analysis
│   │   ├── image_scanner.py        binary + EXIF metadata + OCR text analysis
│   │   ├── email_scanner.py        phishing + attachment analysis
│   │   ├── text_scanner.py         link + script injection analysis (Markdown)
│   │   ├── csv_scanner.py          formula-injection analysis
│   │   ├── xml_scanner.py          XXE + malformed-XML analysis
│   │   ├── html_scanner.py         dangerous markup + event-handler analysis
│   │   └── shortcut_scanner.py     LNK + Internet Shortcut (.url) analysis
│   │
│   └── rules/
│       ├── macro_rules.json        weighted custom detection rules
│       └── yara/
│           ├── vba_autoexec.yar
│           ├── vba_shell_execution.yar
│           ├── vba_download_cradles.yar
│           ├── vba_obfuscation.yar
│           ├── vba_persistence.yar
│           └── vba_process_injection.yar
│
├── tests/                          pytest suite (37 tests)
│   ├── conftest.py                 shared fixtures
│   ├── test_quarantine_dir.py
│   ├── test_cli_quarantine_dir.py
│   ├── test_clamav_scanner.py      ClamAV layer, subprocess mocked per-outcome
│   ├── test_clamav_pipeline_integration.py  ClamAV → verdict, end-to-end
│   └── test_image_ocr.py           OCR layer, synthetic phishing-image fixture
│
├── sample_files/                   test documents (generated by generate_test_samples.py)
├── <quarantine_dir>/               blocked files (created at runtime, only if --quarantine-dir given)
├── clean_output/                   clean files (created at runtime, only if --clean-output-dir given)
└── logs/                           audit_log.jsonl (created at runtime)

---

Adding Detection Rules

Custom rules (no Python required)

Edit docguard/rules/macro_rules.json:

{
  "rules": [
    {
      "name": "My_New_Rule",
      "weight": 40,
      "patterns": ["SuspiciousFunction", "DangerousKeyword"],
      "description": "Detects my new threat pattern."
    }
  ]
}

YARA rules

Add a .yar file to docguard/rules/yara/. It is picked up automatically on the next run — no Python changes needed. Include a weight field in rule metadata:

rule My_New_YARA_Rule
{
    meta:
        description = "Detects something suspicious"
        weight      = 50

    strings:
        $s1 = "SuspiciousString" nocase

    condition:
        any of them
}

---

Generating Test Samples

# Install COM automation support (Windows only, required for .xlsm/.xls/.docm/.doc)
py -3.12 -m pip install pywin32

# Enable VBA project access in Excel and Word:
# File → Options → Trust Center → Trust Center Settings
# → Macro Settings → check "Trust access to the VBA project object model"

py -3.12 generate_test_samples.py

Generates 11 test files covering all supported formats with realistic malicious patterns.

---

Dependencies

Package	Purpose
oletools	VBA macro extraction (olevba)
yara-python	YARA signature matching
XLMMacroDeobfuscator	Excel 4.0 / XLM macro detection
pypdf	PDF structural analysis
Pillow	Image EXIF metadata scanning
openpyxl	XLSX content analysis
extract-msg	Outlook OLE2 .msg parsing
defusedxml	XXE-safe XML parsing (xml_scanner.py)
olefile	OLE2 stream inspection (legacy .doc/.xls/.msg detection)
pytesseract	OCR text extraction from images (image_scanner.py Layer 3)

Two further capabilities depend on a system binary, not a pip package — see Optional system dependencies:

Binary	Purpose
clamscan (ClamAV)	Known-malware signature scanning, cross-cutting across all formats
tesseract (Tesseract OCR)	Backs pytesseract above — required for the OCR layer to actually run

---

Gap Analysis Against validators.py

The company's existing validators.py (used as the synchronous upload gate) was reviewed against DocGuard's checks. The following checks existed in validators.py but not in DocGuard, and have now been added:

Check	Where it was added
Image decompression bomb (pixel count > 50MP)	image_scanner.py
Expanded HTML-injection patterns (vbscript:, onclick=, onmouseover=, data:text/html, <embed>)	text_scanner.py
Script/HTML-injection scan inside OOXML XML content	office_scanner.py
CSV formula-injection scan	new csv_scanner.py
XML XXE / malformed-XML scan	new xml_scanner.py
HTML dangerous-markup scan	new html_scanner.py
Windows shortcut (.lnk) magic-byte + dangerous-target scan	new shortcut_scanner.py
Internet shortcut (.url) dangerous-scheme scan	new shortcut_scanner.py
.eml registered as a distinct routable extension	file_router.py
MSG oversized-file check (50MB)	email_scanner.py

Checks already present in DocGuard that exceed validators.py's equivalent (VBA/YARA macro analysis, PDF structural analysis, co-occurrence heuristics, Chr()/base64 deobfuscation) were left unchanged — validators.py's versions of those checks are a subset of what DocGuard already does.

One structural difference from validators.py remains, but by design rather than by gap: validators.py uses python-magic (libmagic) for its MIME-vs-extension cross-check, while DocGuard's own content_sniffer.py — magic bytes, ZIP-internal [Content_Types].xml inspection, and OLE2 stream-name inspection, see Content-Based Routing above — reaches the same end (route on real content, not the filename) without a libmagic system dependency, which avoids Windows install friction. Both approaches close the same gap; this one just doesn't need a compiled system library to do it.

---

Recent Additions: ClamAV + OCR

Two detection layers were added on top of the original per-format scanner set:

• ClamAV signature scanning (docguard/clamav_scanner.py) — cross-cutting, runs against every file regardless of format. See Cross-Cutting Layers above.
• OCR text scanning (image_scanner.py, Layer 3) — extracts and scans visible text rendered into images. See Cross-Cutting Layers above.

Sandboxing (dynamic execution of macros in an isolated environment) was evaluated and intentionally not pursued at this stage — it would require new execution infrastructure (VM/container orchestration) rather than another static scanner class, and is a larger scope decision than the additions above.

---

Known Limitations

• ViperMonkey (VBA emulation) is not integrated due to a pyparsing version conflict with Python 3.12. It is documented as a future roadmap item.
• The PDF test sample produces a benign incorrect startxref pointer warning from pypdf — this is an artifact of the hand-crafted test file, not a scanner bug.
• generate_test_samples.py requires a Windows machine with Excel and Word installed for the COM-dependent file types (.xlsm, .xls, .docm, .doc). The remaining formats generate on any platform.
• ClamAV signatures: the ClamAV layer has been verified end-to-end against a hand-built, offline test signature (confirmed it correctly flips a verdict to BLOCKED), but has not yet been validated against real-world virus definitions in an environment where freshclam can reach ClamAV's signature CDN. Run freshclam once in your deployment environment and confirm a real detection before relying on this layer in production.
• Sandboxing is not implemented — see Recent Additions above.