hyperplane-eval 0.1.2

A modular framework for evaluating and verifying agentic LLM outputs.

Asymptotic Agent Evaluation Framework

The Asymptotic Agent Evaluation Framework is a Python-based system designed to evaluate AI agents across a global multi-dimensional input space using a linear pipeline architecture. It strategically explores the boundaries of agent reliability to map the "Safe Polytope" of operation.

🚀 Overview

The framework evaluates how AI agents perform as they move through different "dimensions" of complexity (e.g., urgency, ambiguity, complexity). By sampling these dimensions and generating synthetic test cases, the system maps the Safe Polytope — the operational volume where an agent's performance is mathematically reliable ($P_{sat} \ge 0.95$).

Key Features

• Adaptive Navigation: Uses Sobol sequences for uniform expansion from stable regions and binary search to pinpoint failure boundaries.
• Chain-of-Thought (CoT) Evaluation: A reasoning-first judge LLM analyzes compliance before scoring, ensuring high-fidelity boundary detection.
• Geometric Analysis: Calculates the hyper-volume of the agent's safe operational space using N-dimensional Convex Hulls.
• Persistence: Automatically saves the evaluation state to JSON after each iteration for crash-resilience.
• Linear Pipeline: Strictly typed state progression from raw coordinates to final performance metrics.
• Local Execution: Evaluates code directly through AST parsing and local binding execution without WebSocket overhead.
• Provider Agnostic: Uses LiteLLM to dynamically connect to any LLM provider (OpenAI, Gemini, vLLM, etc.) via simple API keys.

🔄 The Linear Pipeline & Iterative Mutation

The evaluation framework operates as a streaming loop that systematically explores the boundary of an agent's failure.

[!NOTE] Algorithm Update: The framework has transitioned from Zero-Shot Synthesis to the Iterative Mutation Pipeline to prevent semantic drift, reduce API overhead, and achieve perfect variable isolation.

graph LR
    A[InputSpace] -->|Origin Prompt| B[Sequential Mutation Engine]
    B -->|Mutated Prompt| C[AgentRunner]
    C -->|Executed Vector| D[AgentOutputEvaluator]
    D -->|Evaluated Vector| E[InputSpace History]
    E -->|Evaluated Vector| F[Adaptive Navigator]
    F -->|Next Coord| B

State Transitions (Core Vector Hierarchy)

The framework uses a strictly typed progression to ensure data integrity:

1. ScenarioVector (Stage 1): Raw N-dimensional mutation coordinates (Urgency, Flooding, Bijection).
2. SynthesizedVector (Stage 2): Origin prompt mutated by the coordinates.
3. ExecutedVector (Stage 3): Mutated prompt run through the target agent (N runs).
4. EvaluatedVector (Stage 4): Performance score ($P_{sat}$) evaluated against the target rubric rule.

🛠 Technology Stack

• Language: Python 3.11+
• Data Modeling: pydantic
• Math/Geometry: numpy, scipy (Sobol sequences, ConvexHull analysis)
• Orchestration: asyncio for parallel scenario generation and execution.
• LLM Integration: litellm for universal API connectivity.

📂 Project Structure

├── engine/                      # Core Evaluation Logic
│   ├── domain/                  # Vector state hierarchy and Prompt features
│   ├── stages/                  # Pipeline stages (input_space, generator, evaluator, navigator, etc.)
│   ├── pipeline.py              # Central evaluation pipeline orchestrator
│   └── prompt_loader.py         # Utility for loading external LLM prompts
├── adapters/                    # External Interfaces
│   ├── llms/                    # Universal LLM Client (LiteLLM Wrapper)
│   ├── runners/                 # Target agent execution abstractions
│   └── local_bindings/          # AST scanning and local function execution
├── cli/                         # Interactive CLI Interface
│   └── app.py                   # CLI Application and real-time dashboard logic
├── prompts/                     # Externalized LLM prompt templates (.txt)
├── reporting/                   # Results Analysis
│   └── analyser.py              # Report compiler & vulnerability analyser
├── results/                     # Evaluated state & reports
└── README.md                    # Project documentation

🚦 Getting Started

Prerequisites

• Python 3.11+
• Access to an LLM inference server or API Key (e.g., OpenAI, Gemini, or a local vLLM instance)

Setup & Installation

1. Environment Setup:

   python3 -m venv .venv
   source .venv/bin/activate
   pip install -r requirements.txt
   

2. Configuration: The framework has an interactive configuration step built-in. Launch the CLI to pick a target python file, select/enter rules, configure scenarios, and supply your chosen LLM model/API Key.

Running the Evaluation

Execute the interactive CLI:

PYTHONPATH=. python3 cli/app.py

The system will:

1. Prompt you for setup configurations in a CLI Wizard.
2. Initialize the search space and extract safety dimensions.
3. Spawn workers to systematically stress-test the agent rules locally.
4. Render a real-time CLI dashboard with a progress bar and events log.
5. Save evolving states to results/ and automatically launch the final HTML results dashboard in your browser.

[!NOTE] The results/ directory is tracked by Git, but its contents (JSON states, HTML reports, logs) are ignored to keep the repository clean.

📈 Analysis Results

The framework outputs a hyper-volume metric (Reliability Coverage) which represents the percentage of the input space where the agent meets the reliability threshold. This allows for direct mathematical comparison between different model versions or system prompts.

📄 License

This project is licensed under the Apache License, Version 2.0. See the LICENSE file for more information.