vexrag 0.1.0

Red Team testing for functional correctness of RAG systems under attack conditions.

VexRAG

A toolkit for assessing the functional correctness of retrieval-augmented generation (RAG) systems under attack conditions.

Sample RAG stacks for getting started: RAG examples.

Quickstart

1) Install

pip install vexrag

For vector DB-specific extras:

pip install "vexrag[qdrant]"
pip install "vexrag[chroma]"
pip install "vexrag[faiss]"

2) Verify installation

vx --help

3) Run a scan from config

vx run --config path/to/scan.yml

Use sample configs from RAG examples/ as a starting point.

Publish to PyPI

Prerequisites

• PyPI account and a project token
• Clean git working tree
• Version bump in pyproject.toml ([project].version)

Build and validate

python -m pip install --upgrade build twine
python -m build
python -m twine check dist/*

Upload

TestPyPI (recommended first):

python -m twine upload --repository testpypi dist/*

Production PyPI:

python -m twine upload dist/*

Set token via env var:

export TWINE_USERNAME="__token__"
export TWINE_PASSWORD="pypi-***"

Project roadmap

Canonical checklist: notes/TODO.md.

Done

• Small RAG (in-memory)
• PoisonedRAG target scan pipeline with core target, scan, and evaluation contracts
• PoisonedRAG CLI scan flow wired from YAML config with multi-context poisoning runs
• Core package facade exports clarified for shared APIs
• StackOverflow XML/TSV to Qdrant ingestion scripts for large dataset indexing
• PoisonedRAG generation improvements: poisoning styles, corpusN payloads, and query-prefixed adversarial outputs
• Automatic attack case generation and consolidated example scan configs
• HijackRAG attack support with CLI generate-cases
• vLLM target/provider support for scan execution
• Core modularization for config/retrieval/runtime

In Progress

• PoisonedRAG hardening: broaden scenario coverage, stabilize metrics, and add end-to-end validation runs
• Medium RAG examples stabilization across vector DB backends and multi-attack eval flow

Next

• Finalize full end-to-end runnable demo for the huge StackOverflow + Qdrant pipeline
• Promote selected wip milestones to stable feature/documented workflow status

Ideas / Backlog

• Red-team testing methods for API-interacting RAG services (local RAG targets)
• Red-team testing methods for the VexRAG CLI (local RAG targets)