LayaEval

Ultra-Fast Neural Evaluation Framework for Agent and RAG Systems
42 production-ready metrics • Sub-50ms latency • Zero external LLM judge costs • Calibrated confidence

Documentation Index • Highlights • Why LayaEval • Installation • Quickstart • Notebooks • Architecture • Metric Catalog (42) • CLI Reference • Acknowledgements • License

Table of Contents

• Key Highlights
• Why LayaEval?
• Installation
  • Standard pip / uv
  • Google Colab & Jupyter Setup
• Quickstart & Real Code Examples
  • 1. Instant Multi-Metric RAG Evaluation
  • 2. Autonomous Agent Execution & Trace Evaluation
  • 3. Direct Callable Metric Objects
  • 4. Managed Evaluator Context Session
  • 5. Dataset Benchmark Evaluation & Reports
  • 6. Infrastructure Detection & Progress Tracking
• Interactive Notebooks
• How It Works & Architecture
• Model Calibration & Best Practices
• Supported Metrics Catalog (42 Metrics)
  • 1. RAG Metrics Catalog (20 Metrics)
  • 2. Agent Metrics Catalog (22 Metrics)
• CLI Usage Reference
• Development & Testing
• Acknowledgements & Special Thanks
• License

LayaEval is a fast, developer-first Python evaluation SDK for Agentic workflows and RAG pipelines, powered by the Laya MCP Server.

LayaEval brings 42 production-grade evaluation metrics (20 RAG metrics + 22 Agent metrics) directly into your CI/CD and production verification loops—executed as calibrated, non-autoregressive neural decisions with zero external LLM API cost and sub-50ms GPU latency.

Key Highlights

• Fast Non-Autoregressive Inference: Replaces slow, multi-second LLM-as-a-judge roundtrips with sub-50ms evaluation on modern GPUs (e.g. NVIDIA CUDA, Apple Silicon Metal) and ~150–450ms on CPU.
• Zero External Token Cost: No OpenAI or Anthropic API bills for running quality guardrails and regression suites. All inference runs on your own local infrastructure.
• 42 Production-Ready Metrics: Comprehensive coverage spanning retrieval quality, groundedness, hallucination risk, multi-step agent traces, tool accuracy, policy compliance, and prompt injections.
• Single Forward Pass Batching: Evaluates multiple metrics for an interaction in a unified forward pass for optimal throughput.
• Calibrated Probabilities & Scores: Outputs statistical probabilities ($P(\text{true})$), confidence intervals, and configurable pass/fail thresholds.
• Ultra-Lightweight Client SDK: Zero heavy dependencies required for the client library (numpy only). Connects to a local or remote laya-mcp-server.
• Two Execution Modes: Supports both live local neural evaluation via Laya MCP Server and fast offline demo simulation (demo=True) for lightweight testing and development without loading neural weights.

Why LayaEval?

Installation

Install via uv (recommended) or pip:

1. Client SDK Only (Ultra-lightweight, NumPy only)

Use this if you already have laya-mcp-server running, or plan to use offline simulation mode (demo=True):

uv add layaeval
# or: pip install layaeval

2. Full Setup with Local Laya MCP Server (Includes PyTorch & neural weights)

Use this to run local neural inference on your machine:

uv add "layaeval[laya]"
# or: pip install "layaeval[laya]"

Google Colab / Jupyter Notebooks:

!pip install "layaeval[laya]"

Quickstart

Execution Modes:

• Live Neural Evaluation (Production): When layaeval[laya] is installed, LayaEval connects to the local non-autoregressive neural model (laya-mcp-server), running genuine semantic evaluations directly on your GPU/CPU with zero external API fees.
• Mock Test Mode (MockMCPClient / demo=True): Designed specifically for CI/CD pipeline integration and offline wiring tests where loading PyTorch and neural model weights is not desired. It returns deterministic mock fixtures.

1. Instant Multi-Metric RAG Evaluation

Evaluate multiple metrics in a single neural forward pass with zero boilerplate:

import layaeval

results = layaeval.evaluate(
    ["groundedness", "hallucination", "response_relevance"],
    query="What is the capital and population of France?",
    retrieved_context="Paris is the capital and most populous city of France, with 2.1M residents.",
    response="Paris is the capital of France, home to about 2.1 million people.",
)

for r in results:
    print(
        f"[{r.metric_name:<20}] Passed: {r.passed:<5} | Value: {r.value:.2f} | Confidence: {r.confidence:.2f}"
    )

Output:

[groundedness        ] Passed: True  | Value: 3.60 | Confidence: 0.49
[hallucination       ] Passed: True  | Value: 0.13 | Confidence: 0.87
[response_relevance  ] Passed: True  | Value: 3.55 | Confidence: 0.45

2. Autonomous Agent Execution & Trace Evaluation

Audit multi-step tool calls, goal achievement, and task completion using metric classes or identifiers:

import layaeval
from layaeval import TaskCompletion, ToolCallAccuracy

task = "Find user #104 and check subscription status."
trace = "Step 1: db.query(user_id=104) -> Plan: Pro\nStep 2: stripe.check(sub_id=sub_104) -> Active"
result = "User #104 has an active Pro subscription."

res = layaeval.evaluate(
    TaskCompletion(threshold=0.50),
    task=task,
    agent_trace=trace,
    final_result=result,
)

print(f"Task Completed: {res.passed} (P(true) = {res.noul:.2f}, confidence = {res.confidence:.2f})")

Output:

Task Completed: True (P(true) = 0.68, confidence = 0.68)

3. Direct Callable Metric Objects

Metric objects can be instantiated with custom thresholds and called directly like standard functions:

from layaeval import Groundedness

groundedness = Groundedness(threshold=3.0)

res = groundedness(
    retrieved_context="Paris is the capital of France.",
    response="Paris is the capital of France.",
)

print(f"Passed: {res.passed} (Score: {res.score:.2f}/4.0 | Confidence: {res.confidence:.2f})")

Output:

Passed: True (Score: 3.17/4.0 | Confidence: 0.24)

4. Managed Evaluator Context

For dedicated server processes, custom models, or high-throughput batch sessions:

from layaeval import Evaluator, Groundedness, HallucinationRisk

with Evaluator(model="auto") as ev:
    results = ev.evaluate(
        [Groundedness(threshold=3.0), HallucinationRisk(threshold=0.2)],
        retrieved_context="Revenue grew by 24% year-over-year.",
        response="The company posted 24% YoY revenue growth.",
    )
    for r in results:
        print(f"{r.metric_name}: {r.passed} (Score/Risk: {r.value:.2f})")

Output:

groundedness: True (Score/Risk: 3.82)
hallucination: True (Score/Risk: 0.00)

5. Dataset Evaluation & Reporting

Run evaluations against entire benchmark datasets and generate structured summary reports:

from layaeval import EvalDataset, EvalReport, evaluate

dataset = EvalDataset.from_jsonl("eval_benchmark.jsonl")

report = evaluate(
    dataset,
    metrics=["groundedness", "hallucination", "response_relevance"],
    name="v1.2-rag-baseline",
)

print(report.summary())
print(f"Pass Rate: {report.pass_rate * 100:.1f}%")
print(f"Total Evaluations: {report.total_evaluations}")

# Save report to JSON for CI/CD gates or archiving
report.save("eval_report.json")

# Reload saved report for regression checks or downstream analysis
loaded_report = EvalReport.load("eval_report.json")

Output:

======================================================================
LayaEval Evaluation Report  (Total: 9, Pass Rate: 100.0%)
======================================================================
Metric                       Type     Count   Pass Rate    Mean Value
----------------------------------------------------------------------
groundedness                 score    3       100.0%       3.80/4.0  
hallucination                noul     3       100.0%       6.00%     
response_relevance           score    3       100.0%       3.80/4.0  
======================================================================
Pass Rate: 100.0%
Total Evaluations: 9

6. Infrastructure Detection & Progress Tracking

LayaEval automatically detects your compute infrastructure (NVIDIA CUDA GPUs, Apple Silicon Metal/MPS, or multi-core CPUs) and displays real-time progress bars as well as internal process visibility (including HuggingFace model weight downloads and MCP server initialization):

import layaeval

# 1. Check your compute accelerator
hw = layaeval.detect_hardware()
print(hw.description)

# 2. Evaluate dataset with real-time progress bar (enabled by default)
report = layaeval.evaluate(dataset, metrics=["groundedness", "hallucination"], progress=True)

Output:

[LayaEval] Hardware: GPU: NVIDIA A100-SXM4-40GB (CUDA, 40.0 GB VRAM, Device 0)
[LayaEval] Initializing Laya MCP Server (model: auto)...
[LayaEval] Evaluating (v1.2-rag-baseline): [████████████████████████] 100% (50/50 examples)

(Note: To silence progress output in CI/CD, set progress=False or export LAYAEVAL_PROGRESS=0)

Interactive Notebooks

Hands-on Jupyter notebooks with end-to-end code, real-world examples, and step-by-step walkthroughs are available in the examples/notebook/ folder:

To launch the notebooks with all dependencies:

# Launch Jupyter Lab directly via uv
uv run --with jupyter jupyter lab examples/notebook/

How It Works

flowchart LR
    subgraph Client["Your Application / Pipeline"]
        A["RAG Context & Response"] --> C["LayaEval SDK"]
        B["Agent Execution Trace"] --> C
    end

    subgraph Batcher["LayaEval Orchestrator"]
        C --> D["Pack Questions & States"]
        D --> E["Single JSON-RPC / MCP Request"]
    end

    subgraph MCP["Laya MCP Server"]
        E --> F["Non-Autoregressive Neural Model"]
        F --> G["Multi-Question Forward Pass"]
    end

    G --> H["Calibrated Verdicts\nScore (0-4) • P(true) • Confidence"]
    H --> C

1. State Aggregation: You supply standard state keys (query, retrieved_context, response, agent_trace, etc.).
2. Batch Construction: LayaEval compiles the requested metrics into typed neural decision prompts.
3. Sub-50ms Inference: The Laya MCP Server runs a non-autoregressive forward pass, returning calibrated probabilities rather than generating arbitrary text tokens.
4. Normalized Results: Every output is mapped to a typed EvalResult with normalized scores, confidence intervals, and pass/fail statuses against your specified thresholds.

Model Calibration & Best Practices

LayaEval uses calibrated probabilities ($P(\text{true})$), prediction confidence scores, and rubric-graded scores (0.0 to 4.0) instead of unstructured free-text generation:

• $P(\text{true})$ Probability: Reflects the model's posterior likelihood that a factual or behavioral condition is met (e.g., that an answer is fully grounded or that a tool call was necessary).
• Confidence: Quantifies the model's certainty in its prediction.
• In-Domain Validation: Out-of-the-box base checkpoints provide a fast, zero-shot baseline for common RAG and agent interactions. For safety-critical or specialized production domains (e.g. medical, legal, or finance RAG):
  1. Curate a Benchmark Set: Collect 100–200 representative labeled examples from your system logs.
  2. Tune Decision Thresholds: Customize metric thresholds to balance precision vs. recall (e.g., use HallucinationRisk(threshold=0.15) for strict safety guardrails).
  3. Establish CI Regression Gates: Save reports with report.save("baseline.json") and run layaeval compare baseline.json candidate.json in your CI/CD pipeline to automatically catch regressions before deployment.

Supported Metrics Catalog

LayaEval uses three calibrated output types:

• Score (0.0 – 4.0): Graded rubric (0: poor, 1: weak, 2: partial, 3: mostly good, 4: perfect). Default pass threshold is $\ge 3.0$.
• Probability $P(\text{true})$ (0.0 – 1.0): Calibrated likelihood for positive qualities. Default pass threshold is $\ge 0.50$ (or $\ge 0.70$).
• Risk Probability $P(\text{true}) \le \text{threshold}$: Calibrated likelihood of negative events (hallucination, contradiction, security injection). Safe pass threshold is $\le 0.20$ or $\le 0.30$.

1. RAG Metrics Catalog (20 Metrics)

Interactive Notebook: Open examples/notebook/rag_metrics_evaluation.ipynb for runnable code examples of each RAG metric.

2. Agent Metrics Catalog (22 Metrics)

Interactive Notebook: Open examples/notebook/agent_metrics_evaluation.ipynb for runnable code examples of each Agent metric.

CLI Usage

LayaEval includes a command-line interface for benchmarking, dataset runs, and regression testing:

# 1. List all 42 available metrics with descriptions
layaeval list-metrics

Output:

Available LayaEval Metrics:

ID         Name                             Type    
----------------------------------------------------
RAG-001    retrieval_relevance              score   
RAG-002    document_relevance               score   
RAG-003    context_sufficient               noul    
...
AG-001     intent                           choice  
AG-002     task_completed                   noul    
...

# 2. Evaluate a JSONL dataset with real-time progress bar
layaeval run dataset.jsonl --metrics groundedness hallucination task_completed --output report.json

# 3. Compare candidate run against a baseline to check for regressions
layaeval compare baseline_report.json candidate_report.json

# 4. Run built-in test suite directly from your terminal
layaeval test

Development & Testing

Run unit tests and lint checks using uv:

# Install dev dependencies
uv sync --all-extras

# Run test suite
uv run pytest

# Check code formatting and linting
uv run ruff check .

Acknowledgements & Special Thanks

Special thanks and gratitude to the Laya open-source contributors and community. LayaEval is powered by the foundational non-autoregressive neural architectures, latency breakthroughs, and high-performance Model Context Protocol (MCP) server engineered by the Laya project.

• Laya GitHub Repository: github.com/NandhaKishorM/laya
• Laya MCP Ecosystem & Docs: github.com/NandhaKishorM/laya/blob/main/docs

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.