hyperplane-eval 0.1.9

Local tool to evaluate AI agents and find their weak points.

Hyperplane Eval

Hyperplane Eval helps teams discover behavioral failures before deployment.

It works as a CLI and programmatic tool for evaluating AI agents against business, security, and ethical requirements using intelligent test generation rather than manually written test cases.

Why Hyperplane?

Most AI teams evaluate agents with a few dozen manually written prompts.

The problem is that manually written tests only cover a tiny fraction of the agent's behavioral space.

Hyperplane automatically explores that space, generating thousands of semantically diverse inputs to uncover failures before users do.

Features

• Generate thousands of semantically diverse test cases
• Evaluate business, security, and ethical requirements of your agent
• Automatically find edge cases and breaking points
• Local-first CLI workflow
• Framework-agnostic agent integration
• Detailed evaluation reports. See example here

CLI Integration

Hyperplane operates entirely locally through a terminal-based orchestration wizard.

Setup & Installation

Install the framework via pip:

pip install hyperplane-eval

Running the CLI

Run the interactive CLI directly in your terminal from inside your project directory:

hyperplane

The CLI wizard will guide you through:

1. Target Binding: Automatically finding your agent's code.
2. Constraint Definition: Setting natural language rules for your agent.
3. Configuration: Setting the scale and depth of the evaluation.
4. Execution: Running the evaluation with a live dashboard.

Hyperplane outputs a structured dataset and an HTML report identifying the specific types and characteristics of inputs that cause the agent to fail, helping you quickly isolate prompt engineering or logic regressions.

Programmatic Integration

You can also integrate Hyperplane Eval directly into your Python codebase or CI/CD testing suites using the Evaluator API:

from hyperplane import Evaluator
from litellm import Router

# 1. Initialize your litellm Router
router = Router(
    model_list=[{
        "model_name": "gpt-4o", 
        "litellm_params": {"model": "gpt-4o"}
    }]
)

# 2. Define your target agent
def my_agent(prompt: str) -> str:
    return "I am a safe agent."

# 3. Initialize the Evaluator
evaluator = Evaluator(
    agent_desc="A helpful AI assistant",
    param_desc={"prompt": "The user input prompt"},
    target_callable=my_agent,
    llm_client=router,
    model_name="gpt-4o"
)

# 4. Add constraints and run
evaluator.run(rules=["Never execute tool calls with unsafe parameters.", "Always respond in English."])

Architecture & Methodology

Evaluating agentic systems presents a curse-of-dimensionality problem due to the infinite input space. Hyperplane mitigates this via a dimensional reduction and bounded sampling approach:

1. Orthogonal Dimension Extraction: The target heuristic or constraint (e.g., a business logic rule) is passed to an LLM, which extracts a set of orthogonal, continuous axes representing the variance in potential inputs (e.g., user_frustration [0,1], budget_constraint [0,1]).
2. Quasi-Random Initialization: The framework maps a bounded multi-dimensional continuous input space $S \in [0, 1]^D$. It utilizes Sobol sequences to generate a low-discrepancy initialization grid to ensure uniform volumetric coverage without clustering.
3. Synthetic Input Generation: Bounded coordinate vectors are mapped back into natural language. A Generator LLM synthesizes adversarial or conversational inputs as structured payloads that strictly adhere to the defined vector coordinates.
4. Response Classification The target agent executes the synthesized inputs. An Evaluator LLM utilizes Chain-of-Thought (CoT) reasoning to classify the agent's response as a pass (1) or fail (0) against the constraint.
5. Surrogate Modeling & Active Search: The framework fits a Random Forest surrogate classifier over the evaluated points to approximate the failure boundary. It utilizes an active search algorithm to sample points near the decision boundary until volumetric saturation is reached, stopping early via dimension-scaled mismatch rate thresholds.

The resulting artifact is a detailed report allowing engineers to identify the specific input themes and characteristics that reliably induce constraint violations.

Privacy & Security (BYOK)

Hyperplane Eval is designed for enterprise privacy using a Bring Your Own Key (BYOK) architecture:

• 100% Local Execution: The orchestrator and test synthesis engine run entirely on your local machine or CI/CD runner.
• No Telemetry or Data Logging: We do not collect product telemetry, execution logs, or source code.
• Direct Vendor Routing: Your API keys are routed strictly to the LLM provider you configure (via litellm) and are never intercepted or proxied.
• Budget Safe: Built-in safeguards and configurable depth/breadth parameters ensure the evaluation pipeline generates high coverage without blowing up your token bill.
• Open Source Auditing: The entire orchestration pipeline is open-source, allowing your security team to fully audit the codebase.

🛠 Technology Stack

• Language: Python 3.10+
• Data Modeling: pydantic
• Math/Geometry: numpy, scipy (Sobol sequences, ConvexHull analysis)
• LLM Integration: litellm for universal API connectivity (OpenAI, Gemini, Anthropic, or any local vLLM).

📄 License

This project is licensed under the Apache License, Version 2.0. See the LICENSE file for more information.