VIPI — Visual Prompt Injection Scanner for Images
Scan images for prompt injections before an AI agent reads them.
A Python CLI and a GitHub Action. Runs fully local or with hosted providers.
Images can carry harmful instructions that people overlook but AI models read. VIPI finds the text in an image, including text that standard OCR misses, and checks it for prompt injections. It scans single images, folders and committed image changes.
- CI/CD and AI code review: scan pull-request images before AI triage tools or coding assistants read them.
- Image ingestion: scan uploaded or collected images before they enter an AI pipeline.
- Agents with image input: check screenshots before an agent decides or acts on them.
Quick start
Requires Python 3.12 and uv.
Local
Images never leave your machine. Be aware that local mode downloads several gigabytes of models during vipi setup and has hardware requirements as stated in Models and hardware.
uv tool install --python 3.12 'vipi-scanner[ocr,detector-local]'
vipi setup --ocr-mode balanced
vipi scan image.png --ocr-mode balanced --output report.json
Choose a profile; use the same one for setup and scan:
vipi setup --ocr-mode {quality|balanced|speed}
vipi scan image.png --ocr-mode {quality|balanced|speed} --output report.json
Cloud
Hosted OCR pipeline and injection detection. The defaults are Google OCR and OpenAI's decision API for injection detection.
Set only the API keys for the providers you use
(keys,
.env example).
uv tool install --python 3.12 'vipi-scanner[ocr]'
vipi setup --modus cloud
vipi scan image.png --modus cloud --output report.json
Choose providers per scan:
vipi scan image.png --modus cloud \
--ocr-provider {google|openai|mistral} \
--decision-provider {openai|jev} \
--output report.json
If vipi is not on PATH, run uv tool update-shell and restart the terminal.
How it works (Overview)
Every scan runs up to three steps. OCR reads the text in the image. If the result is unclear, a visual model compares it with the image; when text is missing, VIPI reads the image again rotated, with more contrast or upscaled. A detector then checks the final text for prompt injections and reports a finding at a score of 80 or more. The mode decides who provides the models: local runs them on your machine, cloud calls the providers' APIs.
Models and memory per mode
| Step | Local quality (default) |
Local balanced |
Local speed |
Cloud |
|---|---|---|---|---|
| 1 · Read text | or Small · Tiny |
or Medium · Tiny |
or Medium · Small |
or |
| 2 · Verify OCR visually | or |
or |
or |
or |
| 3 · Detect injections | or |
or |
or |
or |
| Memory, Apple Silicon or GPU ¹ | 48 GB unified memory (measured) or 24 GB VRAM |
~16 GB unified memory or 16 GB VRAM |
16 GB unified memory (measured) or 16 GB VRAM |
~8 GB RAM no GPU needed |
| Memory, CPU only ¹ | ~48 GB RAM | ~24 GB RAM | ~24 GB RAM | ~8 GB RAM |
¹ Estimates with the defaults; measured were speed on an Apple M2 with 16 GB and
quality on an Apple M4 Pro with 48 GB.
Decision models load unquantized: 16-bit on Apple Silicon and CUDA, 32-bit on CPU.
Clef-Flash 9B needs ~19 GB in 16-bit and ~38 GB in 32-bit. Disk space,
model IDs and pinned revisions are in Models and hardware.
Performance
How well VIPI currently reads text in images and detects prompt injections, measured on the publicly available VIPI evaluation dataset. New images are welcome: they help us test VIPI more thoroughly, and other projects can use the dataset to evaluate their own OCR or injection detection. See contributing to the dataset.
Exact values and models
Development split: 227 injection and 223 benign images, each detector at its default threshold.
| Cloud OCR | Detector | Detection rate | False positive rate | Words recognized |
|---|---|---|---|---|
| Mistral OCR 4.1 | TypeSafe Jev 1.14 | 81.5% | 0.4% | 84.6% |
| Mistral OCR 4.1 | OpenAI GPT-6 Luna (Decisions API) | 85.0% | 4.5% | 84.6% |
| OpenAI GPT-6 Luna | OpenAI GPT-6 Luna (Decisions API) | 83.3% | 4.5% | 84.8% |
| OpenAI GPT-6 Luna | TypeSafe Jev 1.14 | 78.9% | 1.3% | 86.3% |
| Google Cloud Vision | OpenAI GPT-6 Luna (Decisions API, default) | 78.4% | 3.6% | 88.1% |
| Google Cloud Vision | TypeSafe Jev 1.14 | 74.0% | 0.9% | 88.1% |
| Local OCR | Detector | Detection rate | False positive rate | Words recognized |
|---|---|---|---|---|
PP-OCRv6 Medium (quality) |
Clef-Flash 9B (default) | 56.8% | 4.0% | 77.0% |
PP-OCRv6 Medium (quality) |
Prompt Guard 2 86M | 19.4% | 1.8% | 77.0% |
PP-OCRv6 Small (balanced) |
d1 3B | 45.4% | 2.2% | 72.3% |
PP-OCRv6 Small (balanced) |
Prompt Guard 2 86M | 16.7% | 1.8% | 72.3% |
PP-OCRv6 Tiny (speed) |
d1 3B | 43.2% | 2.2% | 70.0% |
PP-OCRv6 Tiny (speed) |
Prompt Guard 2 86M | 18.5% | 2.2% | 70.0% |
Usage
vipi scan label.png screenshot.jpg # single images
vipi scan ./images --verbose --output report.json # folder, print OCR text, save JSON report
vipi scan ./images --exclude 'generated/' --exclude '*.webp'
vipi scan . --diff origin/main --output report.json # only images changed since origin/main
vipi scan image.png --threshold 90 # stricter finding threshold
# other detector: prepare it first, scans never download models
vipi setup --ocr-mode balanced --detector prompt-guard
vipi scan ./images --ocr-mode balanced --detector prompt-guard
vipi scan image.png --modus cloud --detector jev
For permanent exclusions, copy
.vipiignore.example
to .vipiignore.
| Format | Extensions | Notes |
|---|---|---|
| PNG | .png |
Static images only; animated PNGs are rejected |
| JPEG | .jpg, .jpeg |
EXIF orientation is applied |
| WebP | .webp |
Static images only; animated WebPs are rejected |
| SVG | .svg |
Rendered without scripts or external resources; the source text is checked too |
A score at or above the threshold is a finding. The default depends on the detector: 45 for d1, 8 for Clef-Flash and 80 for the others. Size limits, skipped files and what a score does not tell you are in Limitations.
Every option, credential and report field is in the CLI reference.
GitHub Actions
The Action lives in EnXan/vipi-action and is listed on the GitHub Marketplace. It scans the images a pull request adds or changes. Its default cloud mode uses Google Cloud Vision for OCR and OpenAI for decisions:
name: VIPI image scan
on:
pull_request:
branches: [main]
permissions:
contents: read
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0
- uses: EnXan/vipi-action@v1
env:
GOOGLE_CLOUD_API_KEY: ${{ secrets.GOOGLE_CLOUD_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
- uses: actions/upload-artifact@v7
if: always()
with:
name: vipi-report
path: vipi-report.json
Note: Local mode is not recommended on standard GitHub-hosted runners. They have no GPU and too little memory for the local models, so scans can take very long or fail. In our test on a standard runner,
balancedtook about 9 minutes for two images, plus 2 minutes of setup;qualitycould not even load its model. Self-hosted or larger runners with enough memory, ideally with a GPU, can run local mode, but it remains slower than cloud mode.
Other providers are set with ocr-provider and decision-provider and need
their own keys; see the
Action configuration.
Fork PRs normally get no secrets, so their scans cannot reach the providers.
Workbench
The workbench is a visual companion to the CLI that makes it easier to inspect
scan results, compare evaluations and edit the dataset in your browser. It runs
on your machine and uses the same scan pipeline as vipi scan, with local or
cloud settings.
- Scan an image: choose, drop or paste an image. Inspect the verdict, injection probability, recognized text and individual OCR passes, along with scan time and API cost.
- Evaluate: run a configuration on the dataset's development split, compare OCR and detection results with a saved baseline, and download reports. Runs can be stopped and resumed.
- Dataset: add images, edit labels and reference text, and submit your changes as a pull request on Hugging Face.
The workbench is included in the repository, not the PyPI package. With Python 3.12 and uv installed, run:
git clone https://github.com/EnXan/VIPI.git
cd VIPI
uv sync --extra ocr --extra detector-local --extra research --extra test
uv run vipi setup
uv run python -m research.workbench serve
Open localhost:8000. The default is local Quality;
vipi setup downloads its models, so check the
hardware requirements
first. For cloud scans, skip local model setup, configure the
provider API keys
and choose Cloud under Change settings. Cloud scans incur API charges.
For Evaluate and Dataset, download the evaluation dataset from the repository root before starting the workbench:
uv run python -m research.download
See the workbench guide for settings, baseline comparisons and dataset contributions.
Documentation
- CLI reference and platform validation
- Models and hardware
- OCR recovery and measured limits
- Architecture and development
- Research CLI, workbench and datasets
- Decision records
License
VIPI's code, documentation and assets use the Apache License 2.0. Models, dependencies, datasets and provider marks keep their own terms.
Metadata
Release files for vipi-scanner 0.14.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| vipi_scanner-0.14.1.tar.gz | 242.8 kB | Without provenance | Details for vipi_scanner-0.14.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| vipi_scanner-0.14.1-py3-none-any.whl | Python 3 | none | any | Details for vipi_scanner-0.14.1-py3-none-any.whl |
Total release size: 367.2 kB
Release files / vipi_scanner-0.14.1.tar.gz
| Download URL | vipi_scanner-0.14.1.tar.gz |
|---|---|
| Size | 242.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e6d85fa87b87f541e0e14c3fef78369ffc1720dc7ea05f4eb4375ac644d95e57
|
|
BLAKE2b-256 checksum How to use checksums |
b295460634e9d46ec330b42069ade8916719bf2d0974f6c2951a843c6ed1d3fb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / vipi_scanner-0.14.1-py3-none-any.whl
| Download URL | vipi_scanner-0.14.1-py3-none-any.whl |
|---|---|
| Size | 124.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8abf45fbf3aa42e68290e7fd6084f1ab9811904136081ead3857a37e624b1e34
|
|
BLAKE2b-256 checksum How to use checksums |
759278bd042a0e8ce4fc3c4b054a8ac912f11fe1d4801f6264a39fb5b7873621
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for vipi-scanner 0.14.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.