🛡️ LLM Model Security Scanner (llm-scan)
Enterprise-grade, zero-execution static analysis scanner & cryptographic Proof-of-Finding verification engine for open-source Large Language Models (LLMs) distributed across Hugging Face, ModelScope, and local storage.
llm-scan inspects model serialization formats, weights, configuration manifests, tokenizer dictionaries, chat templates, and custom architecture scripts before they are loaded into runtime environments like PyTorch, Hugging Face Transformers, vLLM, Ollama, or llama.cpp.
📌 Table of Contents
- Why LLM Model Security?
- Core Architecture & Zero-Execution Safety
- Key Features
- 💻 Visual Studio Code Extension
- Installation & Quickstart
- CLI Command Reference
- Proof-of-Finding Verification Engine
- Complete 42-Rule Security Catalog
- Regulatory & Security Framework Crosswalk
- Validated Real-World Models
- Python SDK Reference
- CI/CD GitHub Actions Integration
- Threat Model Lifecycle
- Running the Test Suite
- Author & License
🚨 Why LLM Model Security?
Deploying untrusted third-party foundation models introduces severe attack surfaces into enterprise infrastructure:
- Pickle Remote Code Execution (RCE): Legacy PyTorch
.bin,.pt, and.ckptfiles are serialized via Python'spickleengine. Deserialization allows attackers to execute arbitrary system commands (os.system,subprocess, socket reverse shells) the instanttorch.load()is executed. trust_remote_code=TrueExploits: Repositories declaring custom model classes inconfig.json(auto_map) run arbitrary unvetted Python code on the host machine.- Jinja2 SSTI in Chat Templates (CVE-2024-34359): Malicious
tokenizer_config.jsonchat templates can escape the Jinja sandbox via Python reflection (__class__.__mro__[1].__subclasses__()) to achieve arbitrary code execution. - Format Denial of Service (CVE-2023-44469): SafeTensors 8-byte header allocations can be crafted into memory allocation bombs, crashing host inference servers.
- ONNX External Data Traversal (CVE-2024-27318): Crafty ONNX files can write or read outside repository boundaries via path traversal.
- Token Delimiter Hijacking & Trojan Source (CVE-2021-42574): Token ID collisions and unescaped role tags (
<|im_start|>) break prompt boundary isolation, enabling prompt injection. Unicode bidirectional overrides smuggle invisible instructions into vocabularies. - LoRA Adapter Scaling Sabotage & Stealth Layers: Malicious fine-tuned adapters deliberately strip base model safety alignments by setting astronomical scaling factors ($\alpha \gg r$) or injecting Trojan gate weights.
- Credential & Secret Leaks: Model authors frequently leave hardcoded Hugging Face write tokens, OpenAI API keys, AWS credentials, and SSH private keys in files.
🔒 Core Architecture & Zero-Execution Safety
Traditional scanners often import models into Python, which can accidentally trigger the very exploits they seek to find.
llm-scan enforces a strict Zero-Execution Architecture:
- No
torch.load()Execution: Pickles and PyTorch archives are disassembled into opcodes via Python's standardpickletoolswithout instantiating objects or running__reduce__. - AST Parsing for Python Code: Custom modeling files (
modeling_*.py,configuration_*.py) are analyzed purely as Abstract Syntax Trees via Python's nativeastcompiler. - Memory-Mapped Binary Streaming: SafeTensors and GGUF files are checked via 8-byte stream headers and
mmapslices without allocating memory or evaluating data arrays. - Remote Hub Inspection: Scans remote Hugging Face repositories in seconds over the Hugging Face REST API without downloading gigabytes of model weights.
┌──────────────────────────────────────────────┐
│ Model Target (Local Folder / Hub ID) │
└──────────────────────┬───────────────────────┘
│
┌─────────────────────────────────────┴─────────────────────────────────────┐
▼ ▼
[Remote Hub Inspection] [Local Model Filesystem]
- Repository Metadata - Zero-Execution Disassembly
- Typosquatting Check - SafeTensors / GGUF Stream
- Fast Weightless Audit - AST Static Analysis
│ │
└─────────────────────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────────┐
│ 13 Specialized Scanners │
│ (42 Registered Rules) │
└──────────────┬──────────────┘
│
▼
┌─────────────────────────────┐
│ Finding Verification Engine │
│ (Cryptographic SHA-256 Proof)│
└──────────────┬──────────────┘
│
┌───────────────────────────┼───────────────────────────┐
▼ ▼ ▼
[Rich Terminal] [SARIF / JSON] [Interactive HTML / MD]
Color Tables & Proof Cards GitHub Security Tab Upload Client Proof Dossiers
✨ Key Features
- 13 Specialized Scanners & 42 Production Rules: Comprehensive coverage across all major model types.
- Deterministic Proof-of-Finding Engine: Generates mathematical, byte-level evidence to prove vulnerabilities to model authors and security teams.
- Interactive HTML Proof Certificates: Self-contained, zero-dependency HTML certificates with embedded syntax highlighters and SHA-256 fingerprints.
- Compliance Crosswalk Built-in: Instant mapping to OWASP Top 10 for LLMs (2025), MITRE ATLAS, NIST AI RMF (AI 100-1), and EU AI Act (Regulation 2024/1689).
- Remote Hugging Face Hub Auditing: Scan public or private repositories directly using the Hugging Face API without downloading 10GB–100GB weights.
- Native CI/CD Integration: Produces standard SARIF v2.1.0 reports directly compatible with GitHub Code Scanning & Advanced Security.
- Wide Format Support: SafeTensors, PyTorch
.bin/.pt, GGUF, GGML, ONNX, PEFT/LoRA adapters, Hugging Face Tokenizers, and Chat Templates.
💻 Visual Studio Code Extension
Prefer an in-editor IDE experience? Install the official LLM Model Security Scanner for Visual Studio Code:
👉 vscode-llm-model-security-scanner
- 🔴 Real-Time As-You-Write Linter: Flags insecure calls (
os.system,eval(),zlib.decompress, credential leaks) as you write Python code and manifests with red & yellow squiggly underlines. - 🛡️ Remote Import Guardian: As soon as you write
AutoModelForCausalLM.from_pretrained("org/model"),pipeline(..., model="org/model"), orSentenceTransformer(...), the extension automatically audits the Hugging Face repository over the API weightless. - 🔍 Inline CodeLens Badges: Displays instant status above model loading lines (
$(pass) Clean (0/100)or$(alert) [HIGH_RISK] [View Proof]). - 💡 1-Click Quick Fixes (
⌘.): Explain vulnerabilities, open documentation anchors, or generate Proof Dossiers. - 🔬 Interactive Proof Webview: View cryptographic Proof-of-Finding certificates and AST traces directly inside a native VS Code tab.
📦 Installation & Quickstart
Prerequisites
- Python $\ge$ 3.9
- Operating System: macOS, Linux, or Windows (WSL)
Install via pip (Editable / Source)
git clone https://github.com/mesutgungor/llm_model_security.git
cd llm_model_security
pip install -e .
Verify the installation:
llm-scan --help
🛠️ CLI Command Reference
1. Scan Local Models
Scan a complete model directory:
llm-scan scan ./path-to-model-directory
Scan a single model file:
llm-scan scan ./models/pytorch_model.bin
Enable deep weight sampling (checks for NaN/Infinity DoS poisoning and sabotaged layers):
llm-scan scan ./models/my-model --deep
Run in Fast Mode (skips tensor arrays, audits metadata, code, templates, and configurations):
llm-scan scan ./models/my-model --fast
2. Remote Hub Auditing (Zero Weight Download)
Scan remote repositories directly on Hugging Face without downloading weight files:
# Public repository
llm-scan scan meta-llama/Llama-3-8B-Instruct --remote
# Private / Gated repository (requires HF token)
llm-scan scan my-org/custom-model --remote --token hf_yourTokenHere
3. Proof-of-Finding Engine (llm-scan verify)
Generate verifiable proof dossiers containing cryptographic file hashes, exact code locations, and non-destructive reproduction steps:
# Verify local model and render terminal proof cards
llm-scan verify ./path-to-model
# Verify remote Hugging Face model and export self-contained HTML proof certificate
llm-scan verify NousResearch/Hermes-3-Llama-3.1-8B --remote --format html --output hermes-certificate.html
# Export Markdown proof dossier for issue tracking & PRs
llm-scan verify THUDM/chatglm-6b --remote --format markdown --output chatglm-proof-dossier.md
# Export machine-readable JSON proof dossier for security orchestrators
llm-scan verify meta-llama/Llama-3-8B --remote --format json --output proofs.json
4. Output Formats
llm-scan supports flexible output formats:
| Format Flag | Target Use Case | Example Command |
|---|---|---|
terminal (default) |
Human interactive inspection | llm-scan scan ./model |
sarif |
GitHub Advanced Security / IDEs | llm-scan scan ./model --format sarif -o results.sarif |
markdown |
GitHub PR comments / Documentation | llm-scan scan ./model --format markdown -o report.md |
json |
Automation pipelines / SIEM / SOAR | llm-scan scan ./model --format json -o report.json |
html (verify) |
Executive & Consumer Proof Certificates | llm-scan verify <target> --format html -o cert.html |
5. CI/CD Quality Gate
Enforce security thresholds in automated deployment pipelines. If any finding meets or exceeds --max-severity, llm-scan exits with non-zero code 1:
llm-scan scan ./models/production-candidate --max-severity HIGH
Severity Levels: LOW | MEDIUM | HIGH | CRITICAL
6. Utility Commands
# Print all 42 registered security detection rules
llm-scan list-rules
# View the full regulatory & framework crosswalk (OWASP, MITRE ATLAS, NIST, EU AI Act)
llm-scan compliance
# View the 5-phase end-to-end LLM security lifecycle threat model
llm-scan threat-model
🔬 Proof-of-Finding Verification Engine
Security scanners frequently encounter pushback when alerting on model repositories: "Is this really exploitable?" or "How do I reproduce this?"
llm-scan verify solves this problem by generating undeniable, verifiable proof artifacts:
- Cryptographic SHA-256 Fingerprint: Proves the exact file bytes scanned.
- Context Code Window: Extracts the exact offending lines and surrounding context.
- AST Syntax Tree / Opcode Trace: Provides the compiler-level structure of the exploit.
- Step-by-Step Reproduction Guide: Clear, non-destructive reproduction steps for engineers.
- Interactive HTML Proof Certificate: Standalone, zero-dependency HTML file featuring:
- Severity badges & status indicators.
- Interactive verification modal.
- One-click copy for reproduction steps and SHA-256 hashes.
- Regulatory framework tags.
📋 Complete 42-Rule Security Catalog
🏛️ Regulatory & Security Framework Crosswalk
Every rule in llm-scan is systematically mapped to the four principal AI safety and cybersecurity governance frameworks:
🧪 Validated Real-World Models
llm-scan has been validated against production foundation models on the Hugging Face Hub:
| Model Repository | Verdict | Risk Score | Key Findings & Verification |
|---|---|---|---|
Qwen/Qwen2.5-0.5B-Instruct |
SAFE |
0 / 100 | Zero findings. Native SafeTensors format, clean tokenizer, no arbitrary remote code. |
gpt2 |
MEDIUM_RISK |
9 / 100 | Contains legacy PyTorch pickle weights (pytorch_model.bin). Flagged for modernization. |
tiiuae/falcon-7b |
HIGH_RISK |
45 / 100 | Declares auto_map in config.json requiring trust_remote_code=True. Verified via AST proof. |
THUDM/chatglm-6b |
HIGH_RISK |
92 / 100 | 5 high-severity findings: base64 decompression at quantization.py:L30, process spawning, auto_map. |
NousResearch/Hermes-3-Llama-3.1-8B |
HIGH_RISK |
45 / 100 | Role delimiter `< |
🐍 Python SDK Reference
Integrate llm-scan directly into your data pipelines, model ingestion workflows, or PyTorch inference servers:
from llm_security_scanner import ModelSecurityScanner, ScannerConfig, SecurityVerdict
# Configure the security scanning engine
scanner = ModelSecurityScanner(
config=ScannerConfig(
scan_pickles=True,
scan_remote_code=True,
scan_chat_templates=True,
scan_safetensors=True,
scan_special_tokens=True,
deep_weight_sampling=True,
)
)
# Audit a local model directory or file
report = scanner.scan("./models/llama-3-8b")
print(f"Model: {report.model_path}")
print(f"Security Verdict: {report.verdict.value}")
print(f"Risk Score: {report.risk_score} / 100")
# Enforce security gates before model loading
if report.verdict in (SecurityVerdict.HIGH_RISK, SecurityVerdict.CRITICAL):
print("🚨 Refusing to load model due to critical security findings:")
for finding in report.findings:
print(f" - [{finding.severity.value}] {finding.title} ({finding.file_path})")
print(f" Remediation: {finding.remediation}\n")
raise SystemExit("Pipeline aborted: Model failed security gate.")
# Safe to proceed with torch.load or Transformers
print("✅ Security gate passed: Model verified safe for production loading.")
🔄 CI/CD GitHub Actions (SARIF Integration)
Integrate llm-scan into your GitHub CI/CD workflows. Security alerts will automatically appear in your repository's GitHub Security > Code Scanning Alerts tab:
name: LLM Model Security Audit
on:
pull_request:
paths:
- 'models/**'
push:
branches: [main]
jobs:
model-security-audit:
name: Model Security Scan & SARIF Upload
runs-on: ubuntu-latest
steps:
- name: Checkout Repository
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'
- name: Install Scanner
run: |
pip install --upgrade pip
pip install -e .
- name: Execute Model Security Scan
run: |
llm-scan scan ./models/candidate-model \
--format sarif \
--output results.sarif \
--max-severity HIGH
- name: Upload SARIF to GitHub Security Tab
uses: github/codeql-action/upload-sarif@v3
if: always()
with:
sarif_file: results.sarif
🌐 Threat Model Lifecycle
llm-scan is architected around the full 5-stage Machine Learning Security Lifecycle:
Phase 1: Training Data Collection
├── Web scraping cache poisoning & Clean-label Trojan injections
└── PII / Secret leakage in pre-training corpora
Phase 2: Training & Model Architecture
├── Malicious custom CUDA / Triton kernels
├── Sleeper Agent backdoor weights & Trojan gates
└── LoRA safety-alignment stripping
Phase 3: Serialization & Distribution (Core Target of llm-scan)
├── Weaponized pickle bytecode RCE (torch.load)
├── trust_remote_code=True auto_map dynamic execution
├── Jinja2 Chat Template SSTI (CVE-2024-34359)
├── SafeTensors & GGUF header memory allocation bombs
├── ONNX external data path traversal (CVE-2024-27318)
├── Tokenizer role delimiter hijacking & Unicode Bidi smuggling
└── Disguised ELF / PE binaries & hardcoded credentials
Phase 4: Deployment & Infrastructure
├── Inference engine memory corruption (vLLM, llama.cpp)
└── GPU memory exhaustion via oversized token generation configs
Phase 5: Post-Deployment & Inference
├── Direct & Indirect Prompt Injections
└── Jailbreaking & Autonomous Agent tool abuse (SSRF)
Run llm-scan threat-model in your terminal to view details on every lifecycle phase.
🧪 Running the Test Suite
The test suite covers all 13 scanner modules, the verification engine, and SARIF export:
# Run all unit and integration tests
python3 -m unittest discover -s tests -v
Expected output:
Ran 35 tests in 0.030s
OK
All 35 tests execute without executing dangerous bytecode or downloading external files.
👤 Author & Contact
Mesut Güngör
- GitHub: @mesutgungor
- Repository: https://github.com/mesutgungor/llm_model_security
📄 License
This project is licensed under the Apache License 2.0. See the LICENSE file for complete terms and copyright notices.
1 maintainer
Metadata
1 maintainer
Release files for llm-model-security-scanner 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llm_model_security_scanner-1.0.0.tar.gz | 88.8 kB | With provenance | Details for llm_model_security_scanner-1.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llm_model_security_scanner-1.0.0-py3-none-any.whl | Python 3 | none | any | Details for llm_model_security_scanner-1.0.0-py3-none-any.whl |
Total release size: 177.2 kB
Release files / llm_model_security_scanner-1.0.0.tar.gz
| Download URL | llm_model_security_scanner-1.0.0.tar.gz |
|---|---|
| Size | 88.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
74a1fc456874a0861b20670a6c4c60ae018f05ffe5b40215c8832ba370c2ec75
|
|
BLAKE2b-256 checksum How to use checksums |
420de2e694168dd25134ff1dec206cf7ae277b937a11ff82b05ccda8013c3a81
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 25, 2026.
Transparency logRelease files / llm_model_security_scanner-1.0.0-py3-none-any.whl
| Download URL | llm_model_security_scanner-1.0.0-py3-none-any.whl |
|---|---|
| Size | 88.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6dee93f6760ce5db156893446adca838ecfa16cd950a5cb164e3d88c9d3fe85c
|
|
BLAKE2b-256 checksum How to use checksums |
8c07ef8c0029e0a65d81a60c72101085320e7df02e64ff7576995f766eaa1b1f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 25, 2026.
Transparency logSecurity for llm-model-security-scanner 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for llm_model_security_scanner-1.0.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 25, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
