Vectory

Precision LLM and agent evaluation - An open-source toolkit for custom benchmarks, AI judges, deterministic metrics, RAG scoring, human review, and proof-grounded trace validation.

Features

• 📊 Dataset Management - Upload JSON, CSV, or PDF files. Automatic column detection and mapping.
• 🤖 LLM-as-Judge - Run critique-first binary judges by default, import promoted failure-mode evaluators, and retain exploratory 1–5 scoring for compatibility.
• 📏 Rule-Based Metrics - Calculate BLEU, ROUGE, exact match, Levenshtein similarity, and regex patterns.
• 🔍 Error Analysis - Review diverse trace samples, capture expert Pass/Fail critiques, discover failure modes, find related examples, and promote focused evaluators.
• 🔎 RAG Metrics - Evaluate retrieval precision, recall, MRR, NDCG, context relevance, faithfulness, and answer completeness.
• 👤 Human Evaluation - Rate outputs manually with customizable scales and criteria.
• 🏆 MTEB Leaderboard - View embedding model benchmarks from the Massive Text Embedding Benchmark.
• 🧪 Custom Benchmarks - Prepare organization datasets and run optional local embedding model comparisons.
• 🧭 Vectory Benchmark - Score submitted agent traces for task success, reality sampling, tool discipline, recovery, control, turn efficiency, and proof grounding.
• ⚙️ Configurable Themes - Switch between Dark, Light, and Colorblind-safe themes for accessibility.

Quick Start

pip install vectoryai
vectory app

Use pipx when you want an isolated CLI install:

pipx install vectoryai
vectory app

The app opens in your browser at http://localhost:8501.

Create a portable failure-discovery workspace directly from AI outputs or traces:

vectory discover traces.jsonl \
  --workspace eval-review \
  --sample-size 30 \
  --dimension-fields feature scenario persona

After expert review and taxonomy acceptance, promote and validate a focused evaluator:

vectory promote eval-review "Unsupported claim" --errors-reviewed
vectory split-labels judge-labels.jsonl \
  --label-column human_label \
  --judge-column judge_label \
  --out judge-splits
vectory validate-judge judge-splits/test.jsonl \
  --human-column human_label \
  --judge-column judge_label \
  --split-manifest judge-splits/split_manifest.json \
  --out judge-validation.json

See the human-grounded evaluation workflow for the app flow, artifact schema, validation metrics, and CI behavior.

Run the included agent benchmark example from the CLI:

vectory benchmark data/vectory_benchmark/example_submission.json

Run the included proof-grounding example:

vectory benchmark data/vectory_benchmark/example_proof_submission.json

Run the policy proof validation example:

vectory benchmark data/vectory_benchmark/example_policy_proof_submission.json

Run the matching policy proof failure example:

vectory benchmark data/vectory_benchmark/example_policy_proof_failure_submission.json

Write an inspectable benchmark report bundle:

vectory benchmark data/vectory_benchmark/example_policy_proof_submission.json \
  --report-out /tmp/vectory-report

Use Vectory as a CI release gate:

vectory gate data/vectory_benchmark/example_policy_proof_submission.json \
  --min-score 0.90 \
  --block-severity critical \
  --report-out /tmp/vectory-gate-report

Installation

Local CLI

Use pip inside an activated virtual environment or project environment:

pip install vectoryai
vectory app

Use pipx when you want Vectory as a standalone local command without mixing its dependencies into another Python project:

pipx install vectoryai
vectory app --port 8501

Source Checkout

git clone https://github.com/tacticaledge/vectory.git
cd vectory
pip install -e ".[dev]"
vectory app

Prerequisites

• Python 3.10 or higher
• pip, pipx, or uv

Contributor Installation

# Create a virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install the project in editable mode
pip install -e ".[dev]"

# Launch the app
vectory app

Optional: Formal Checker Support

The default install does not execute local formal checkers. Proof artifacts submitted in traces are still scored deterministically. Install the optional formal extra only when you want Vectory to run trusted suite-defined checker commands locally:

pip install "vectoryai[formal]"

Formal runtime is disabled by default. Enable it explicitly and provide a local workspace:

vectory benchmark path/to/submission.json \
  --suite path/to/manifest.json \
  --workspace path/to/workspace \
  --allow-formal-runtime

Uploaded submissions cannot provide executable commands. Vectory only runs checker commands defined in the trusted local suite manifest.

Optional: Embedding Benchmark Support

The default install does not include PyTorch. Embedding comparison features are optional because they add large model dependencies. Install them only when you need local embedding benchmark runs:

pip install "vectoryai[embedding]"

For a source checkout, install the optional extra from the checkout:

pip install -e ".[embedding]"

For an existing pipx install, add the optional runtime to the isolated Vectory environment:

pipx inject vectoryai "torch>=2.1.0" mteb sentence-transformers

Usage

1. Upload Your Data

Navigate to the 📊 Dataset page and upload your evaluation data:

• JSON/JSONL: Arrays of objects with input/output/expected fields
• CSV: Tabular data with headers
• PDF: Extracted to markdown (optimized for LLM consumption)

The app will automatically detect column roles, or you can map them manually.

2. Run Evaluations

Choose your evaluation method:

LLM-as-Judge (🤖)

• Start with a critique-first binary Pass/Fail decision
• Load a focused evaluator promoted from observed failure modes
• Select OpenAI or Anthropic and run batch predictions
• Validate predictions against held-out domain-expert labels before release gating

Rule-Based Metrics (📏)

• Select metrics (BLEU, ROUGE, exact match, etc.)
• Run evaluation
• View per-sample and aggregate results

Error Analysis (🔍)

• Review a coverage-oriented sample of complete traces on one screen
• Capture required critiques from the principal domain expert with binary verdicts
• Group notes into a structured failure-mode taxonomy
• Generate AI-assisted taxonomy suggestions from reviewer notes
• Find related traces for human confirmation and analyze reviewed outcomes by product dimension
• Promote failure modes into draft binary evaluators
• Export annotations, taxonomies, evaluators, and a portable review bundle

Human Evaluation (👤)

• Navigate through samples
• Rate on customizable scales
• Add feedback and criteria tags
• Export annotations

Vectory Benchmark (🧭)

• Upload agent run traces as JSON or JSONL
• Score submitted runs against a built-in agent reliability manifest
• Review pathologies such as premature planning, retrieval thrashing, ignored evidence, approval bypass, open proof obligations, failed checkers, and placeholder proofs
• Test policy proof traces where final claims must connect to policy rules, source references, proof obligations, and checker results
• Export run scores, leaderboard summaries, benchmark cards, claim-evidence tables, checkpoint timelines, and static HTML report bundles
• Use CI gates to block releases when scores fall below threshold or critical pathologies appear

Vectory Benchmark is designed for teams that need to understand agent behavior from real traces: whether the agent observed before acting, used tools productively, recovered from failure, respected scope, and stopped before extra turns became low-value work. For private evaluations of internal agents and workflows, use the Vectory contact form: https://vectoryai.com/#contact

Run the included example from the command line:

vectory benchmark data/vectory_benchmark/example_submission.json

Run a proof-grounded policy sample:

vectory benchmark data/vectory_benchmark/example_policy_proof_submission.json

This sample shows why proof grounding is different from ordinary LLM-as-judge, keyword checks, or RAG metrics. The final answer only receives full credit when claims link to evidence, required proof_obligations are closed, policy checker_results pass, and NO_TRANSLATIONS warnings are preserved instead of silently treated as proved facts.

Use the failure fixture to see the expected negative behavior:

vectory benchmark data/vectory_benchmark/example_policy_proof_failure_submission.json

The bad trace claims success despite failed checkers, open obligations, and unsupported evidence, so Vectory should flag pathologies such as unclosed_proof_obligation, ignored_checker_result, policy_regression, and evidence_does_not_support_claim.

Export scores and leaderboard summaries:

vectory benchmark data/vectory_benchmark/example_submission.json \
  --scores-out vectory-scores.json \
  --leaderboard-out vectory-leaderboard.csv

Export a static report bundle:

vectory benchmark data/vectory_benchmark/example_policy_proof_submission.json \
  --report-out /tmp/vectory-report

The report bundle includes index.html, scores.json, leaderboard.json, pathologies.json, claim_evidence_table.json, checkpoints.json, and benchmark_card.json.

Run a release gate:

vectory gate data/vectory_benchmark/example_policy_proof_submission.json \
  --min-score 0.90 \
  --block-severity critical

The gate exits non-zero when a run falls below the threshold, fails its task gate, exceeds the optional pathology-risk limit, or includes a pathology at or above the configured severity.

3. Export Results

Download your evaluation results as CSV or JSON from any results page.

Configuration

API Keys

Configure API keys in one of three ways:

Option 1: Streamlit Secrets (Recommended for deployment)

Create .streamlit/secrets.toml:

[api_keys]
openai = "<openai-api-key>"
anthropic = "<anthropic-api-key>"

Option 2: Environment Variables

export OPENAI_API_KEY="<openai-api-key>"
export ANTHROPIC_API_KEY="<anthropic-api-key>"

Option 3: Manual Input

Enter your API key directly in the UI (stored only in session).

Themes & Accessibility

Vectory includes multiple themes to suit different preferences and accessibility needs. Change your theme in the ⚙️ Settings page.

Available Themes:

Theme	Description
Dark Developer	Modern dark theme optimized for developers (default)
Light	Clean light theme for bright environments
High Contrast (Colorblind Safe)	Blue-orange palette distinguishable by most forms of color blindness
High Contrast Dark	Maximum contrast for users with low vision

The colorblind-safe theme uses a carefully selected palette that works for:

• Deuteranopia (red-green color blindness)
• Protanopia (red color blindness)
• Tritanopia (blue-yellow color blindness)

Streamlit Configuration

The default theme is configured in .streamlit/config.toml:

[theme]
primaryColor = "#6366f1"
backgroundColor = "#0f172a"
secondaryBackgroundColor = "#1e293b"
textColor = "#f1f5f9"
font = "monospace"

Deployment

Python Package Release

Vectory is packaged for Python distribution as vectoryai and exposes the vectory command. Release publishing to PyPI is handled by the Publish Python Package GitHub Actions workflow when a version tag such as v1.0.0 is pushed, or when the workflow is run manually.

Publishing uses PyPI Trusted Publishing. Internal package mirrors, when enabled, use GitHub Actions secrets and variables that are not committed to the public repository.

Docker

FROM python:3.10-slim

WORKDIR /app
COPY . .

RUN pip install .

EXPOSE 8501

CMD ["vectory", "app", "--address", "0.0.0.0"]

Manual Deployment

vectory app --port 8501 --address 0.0.0.0

Project Structure

vectory/
├── app.py                    # Main entry point
├── requirements.txt          # Python dependencies
├── pages/                    # Streamlit pages
│   ├── 1_📊_Dataset.py      # Dataset upload & config
│   ├── 2_🤖_LLM_Judge.py    # LLM-as-Judge evaluation
│   ├── 3_📏_Metrics.py      # Rule-based metrics
│   ├── 4_👤_Human_Eval.py   # Human evaluation
│   ├── 5_🏆_Leaderboard.py  # MTEB leaderboard
│   ├── 6_🧪_Custom_Benchmark.py
│   ├── 7_⚙️_Settings.py     # Theme & API key settings
│   ├── 8_🔍_Error_Analysis.py
│   └── 9_🧭_VectoryBenchmark.py
├── components/               # Core modules
│   ├── __init__.py          # Version and branding
│   ├── models.py            # Pydantic data models
│   ├── themes.py            # Configurable theme system
│   ├── document_loader.py   # File loading utilities
│   ├── error_discovery.py   # AI-assisted taxonomy helpers
│   ├── ui.py                # Animated UI components
│   ├── mteb_evaluator.py    # Embedding evaluation
│   └── evaluators/          # Evaluation implementations
│       ├── llm_judge.py
│       ├── rag_metrics.py
│       └── metrics.py
├── tests/                    # Test suite
└── data/                     # Sample datasets

Testing

# Install test dependencies
pip install -e ".[dev]"

# Run tests
pytest tests/ -v

# Run with coverage
pytest tests/ -v --cov=components --cov-report=html

Troubleshooting

PyTorch Version Error

If you see module 'torch' has no attribute 'compiler':

pip install -e ".[embedding]"

The app works without PyTorch. Local embedding model execution stays disabled until the optional embedding extra is installed.

NLTK Data Missing

If you see NLTK-related errors:

python -c "import nltk; nltk.download('punkt'); nltk.download('punkt_tab')"

API Key Issues

• Ensure your API key is valid and has sufficient credits
• Check that the key is correctly formatted (no extra spaces)
• For Anthropic, ensure you're using the correct model names

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Streamlit - The amazing framework powering this app
• MTEB - Massive Text Embedding Benchmark
• PyMuPDF4LLM - PDF to Markdown conversion
• HuggingFace Evaluate - Evaluation metrics

---

Vectory - Precision LLM Evaluation | Tactical Edge