Causal Agent Verifier (CAV)

Causal Agent Verifier (CAV) is a first-principles evaluation framework for LLM agents.

Unlike other evaluation frameworks that rely on brittle "expected trajectories" or subjective "LLM-as-a-judge" heuristics to grade an agent's reasoning, CAV evaluates agents by auditing their telemetry against Causal Invariants (deterministic rules) and conducting Counterfactual Analysis on critical state mutations.

Why CAV?

LLM agents are difficult to evaluate because they operate autonomously over long horizons, and their trajectories are non-deterministic. A single mistake compounds over time.

CAV solves this by evaluating agents at 3 distinct causal layers:

1. Epistemics (Belief vs Reality): Did the agent hallucinate, or did it perceive the environment correctly?
2. Invariants (Deterministic Rules): Did the agent's actions violate any core system rules (e.g., "Never delete a production database")?
3. Counterfactuals (Causal Rationality): Given what the agent believed, was its action the safest and most rational choice?

Features

• OpenTelemetry / OpenInference Native: Seamlessly ingests standard OTel traces. Works out-of-the-box if your agent (LangGraph, CrewAI, LlamaIndex, Pydantic AI) uses OpenInference instrumentation.
• Sandboxed Execution: Deterministic invariants are executed securely using simpleeval, blocking malicious or arbitrary code execution.
• Multi-Provider LLM Support: Use Gemini, OpenAI, or Anthropic for Layer 3 Counterfactual analysis and Invariant Generation.

Making Your Agent Compatible

CAV relies on OpenTelemetry / OpenInference for standard telemetry. You do not write the agent_trace.json manually—it is automatically generated by your agent framework when it runs (e.g., using openinference-instrumentation-langchain or openinference-instrumentation-openai).

How to Generate agent_trace.json Locally (Without Modifying Prod Code)

Hardcoding an InMemorySpanExporter inside your application code is an anti-pattern, because you would have to rewrite your telemetry configuration when you move to production (where you'd use an OTLPSpanExporter to send traces to a backend collector).

Instead, your application code should remain completely agnostic to where traces are sent. Just instrument the framework:

# app.py (Your production code)
from openinference.instrumentation.openai import OpenAIInstrumentor
OpenAIInstrumentor().instrument()

def run_agent(prompt: str):
    # Agent logic here...
    pass

To capture the traces locally or in your CI/CD pipelines for CAV evaluation, use CAV's capture_traces context manager in your test suite. This dynamically intercepts the OTel traces during the test execution and dumps them to a JSON file, leaving your production code completely untouched.

# test_agent.py (Your evaluation suite)
from app import run_agent
from cav.testing import capture_traces

def test_agent_deployment():
    # 1. Dynamically intercept traces just for this run
    with capture_traces("agent_trace.json"):
        run_agent("Deploy the web app")
        
    # 2. Now pass agent_trace.json into CAV...

However, to use CAV's Epistemics (Belief vs Reality) Layer, you must make two small additions to your evaluation pipeline:

1. Emit Agent Belief

Ensure your agent code emits a custom span attribute called agent.belief. This allows CAV to evaluate what the agent thought was happening.

Schema:

{
  "observed_facts": {"env": "test", "user_auth": "admin"},
  "reasoning": "Since I am an admin in the test env, I will proceed."
}

Prompting Tip: To get your agent to emit this, simply add this instruction to its system prompt:

"Before taking any action, you must output a 'thought' JSON block containing 'observed_facts' (key-value pairs of your current environment state) and 'reasoning' (your rationale)."

Extract that JSON in your agent loop and attach it to the OpenInference trace as the agent.belief attribute.

2. Provide a Ground Truth File (Optional)

To use the Epistemics (Belief vs Reality) layer, you need a ground_truths.json file mapping each agent step_id to the actual state of the environment.

You do not have to write this structure manually. CAV provides a utility to generate a skeleton based on your trace:

from cav.utils import generate_ground_truth_skeleton

# This reads your trace and creates a skeleton ground truth file
generate_ground_truth_skeleton(
    trace_filepath="agent_trace.json", 
    output_filepath="ground_truths.json"
)

The generated file will look like this, ready for you to fill in the true values:

{
  "span_001": {
    "_comment": "Enter ground truth environment variables for step at 2026-09-02T10:00:00Z",
    "env": "prod",
    "user_auth": "guest"
  }
}

If this file is not provided, the Epistemics check is gracefully skipped.

Installation

Install CAV via pip:

pip install causal-agent-verifier

To install with specific LLM provider support:

pip install causal-agent-verifier[gemini]
# or
pip install causal-agent-verifier[openai]
# or
pip install causal-agent-verifier[all]

Quick Start

from cav.engine import CavEngine
from cav.invariants.checker import InvariantChecker
from cav.epistemics import EpistemicEvaluator
from cav.counterfactuals import CounterfactualEvaluator
from cav.telemetry import TelemetryParser

# 1. Initialize Evaluators
checker = InvariantChecker("invariants.json")
epistemics = EpistemicEvaluator(strict_mode=True)
counterfactuals = CounterfactualEvaluator()

# 2. Setup the Engine
engine = CavEngine(
    invariant_checker=checker,
    epistemic_evaluator=epistemics,
    counterfactual_evaluator=counterfactuals,
    goal="Deploy the web application"
)

# 3. Parse your OpenInference Agent Trace and Ground Truth
# The agent trace is generated by your agent framework.
# The ground truth file is generated by your evaluation test suite.
trajectory, ground_truths = TelemetryParser.parse_file(
    trace_filepath="agent_trace.json",
    ground_truth_filepath="ground_truths.json"
)

# 4. Evaluate the Trajectory
report = engine.evaluate_trajectory(trajectory, ground_truths)

# 5. Output Results
from cav.reporters import VerifierReporter
print(VerifierReporter.generate_markdown_report(report))

Evaluating at Scale (Test Suites & CI/CD)

If you have many edge cases or variants to test, you integrate CAV directly into your development cycle (e.g., using pytest):

1. Run your Agent against Test Cases: Execute your agent against a suite of 50 different prompts/scenarios. This will generate 50 separate agent_trace.json files (or one large trace with 50 parent spans).
2. Generate/Maintain Ground Truths: Use the generate_ground_truth_skeleton utility to scaffold the true states for those 50 runs. You only need to do this once when creating the test suite. As long as your test environments are deterministic, you can reuse these ground truth files for all future runs.
3. Automate CAV: Write a simple test script that loops over your 50 traces, passes them to CavEngine, and asserts that report["goal_reached"] == True. If an agent regression causes an invariant to fail, your CI/CD pipeline will catch it immediately.

Advanced Customization

Generating Invariants

CAV provides scaffolding to pre-generate deterministic invariants using an LLM.

from cav.invariants.generator import InvariantGenerator
from cav.llm_providers import GeminiProvider

provider = GeminiProvider(api_key="YOUR_API_KEY")
generator = InvariantGenerator(llm_client=provider)

template = generator.generate_template(
    goal="Clean up test databases",
    context="Only drop tables in the test environment."
)

License

MIT License