LayaEval

Ultra-Fast Neural Evaluation Framework for Agent and RAG Systems
42 production-ready metrics • Sub-50ms latency • Zero external LLM judge costs • Calibrated confidence

Highlights • Why LayaEval • Installation • Quickstart • Notebooks • How It Works • Metric Catalog • CLI Reference • Acknowledgements • License

LayaEval is an ultra-fast, developer-first Python evaluation SDK for Agentic workflows and RAG pipelines, powered by the Laya MCP Server.

LayaEval brings 42 production-grade evaluation metrics (20 RAG metrics + 22 Agent metrics) directly into your CI/CD and production verification loops—executed as calibrated, non-autoregressive neural decisions with zero external LLM API cost and sub-50ms latency.

Key Highlights

• ⚡ Sub-50ms Latency: Non-autoregressive neural verification replaces slow, multi-second LLM-as-a-judge roundtrips.
• 💰 Zero External Token Cost: No OpenAI or Anthropic API bills for running quality guardrails and regression suites.
• 🎯 42 Production-Ready Metrics: Comprehensive coverage spanning retrieval quality, groundedness, hallucination risk, multi-step agent traces, tool accuracy, policy compliance, and prompt injections.
• 🚀 Single Forward Pass Batching: Evaluates multiple metrics for an interaction in a unified forward pass for optimal throughput.
• 📊 Calibrated Confidence: Outputs statistical probabilities ($P(\text{true})$), standard confidence intervals, and configurable pass/fail thresholds.
• 🔌 Native MCP Integration: Direct connection via stdio JSON-RPC or embedded runtime to the Laya MCP ecosystem.
• 🪶 Ultra Lightweight: Zero heavy dependencies required at install time (numpy only).

Why LayaEval?

Installation

Install via uv (recommended) or pip:

# Core package (ultra-lightweight, connects to existing laya-mcp-server)
uv add layaeval

# Or with full local Laya runtime included:
uv add "layaeval[laya]"

Using standard pip:

pip install layaeval
# Or with local Laya runtime & MCP server:
pip install "layaeval[laya]"

Google Colab / Jupyter Notebooks:

!pip install "layaeval[laya]"

Quickstart

1. Instant Multi-Metric RAG Evaluation

Evaluate multiple metrics in a single neural forward pass with zero boilerplate:

import layaeval

results = layaeval.evaluate(
    ["groundedness", "hallucination", "response_relevance"],
    query="What is the capital and population of France?",
    retrieved_context="Paris is the capital and most populous city of France, with 2.1M residents.",
    response="Paris is the capital of France, home to about 2.1 million people.",
)

for r in results:
    print(
        f"[{r.metric_name:<20}] Passed: {r.passed:<5} | Value: {r.value:.2f} | Confidence: {r.confidence:.2f}"
    )

2. Autonomous Agent Execution & Trace Evaluation

Audit multi-step tool calls, goal achievement, and task completion using metric classes or identifiers:

import layaeval
from layaeval import TaskCompletion, ToolCallAccuracy

task = "Find user #104 and check subscription status."
trace = "Step 1: db.query(user_id=104) -> Plan: Pro\nStep 2: stripe.check(sub_id=sub_104) -> Active"
result = "User #104 has an active Pro subscription."

res = layaeval.evaluate(
    TaskCompletion(threshold=0.75),
    task=task,
    agent_trace=trace,
    final_result=result,
)

print(f"Task Completed: {res.passed} (P(true) = {res.noul:.2f}, confidence = {res.confidence:.2f})")

3. Direct Callable Metric Objects

Metric objects can be instantiated with custom thresholds and called directly like standard functions:

from layaeval import Groundedness

groundedness = Groundedness(threshold=3.0)

res = groundedness(
    retrieved_context="Paris is the capital of France.",
    response="Paris is the capital.",
)

print(f"Passed: {res.passed} (Score: {res.score}/4.0 | Reason: {res.reasoning})")

4. Managed Evaluator Context

For dedicated server processes, custom models, or high-throughput batch sessions:

from layaeval import Evaluator, Groundedness, HallucinationRisk

with Evaluator(model="auto") as ev:
    results = ev.evaluate(
        [Groundedness(threshold=3.0), HallucinationRisk(threshold=0.2)],
        retrieved_context="Revenue grew by 24% year-over-year.",
        response="The company posted 24% YoY revenue growth.",
    )
    for r in results:
        print(f"{r.metric_name}: {r.passed} (Score/Risk: {r.value:.2f})")

5. Dataset Evaluation & Reporting

Run evaluations against entire benchmark datasets and generate structured summary reports:

from layaeval import EvalDataset, evaluate

dataset = EvalDataset.from_jsonl("eval_benchmark.jsonl")

report = evaluate(
    dataset,
    metrics=["groundedness", "hallucination", "response_relevance"],
    name="v1.2-rag-baseline",
)

print(f"Pass Rate: {report.pass_rate * 100:.1f}%")
print(f"Total Evaluations: {report.total_evaluations}")
report.save("eval_report.json")

Interactive Notebooks

Hands-on Jupyter notebooks with end-to-end code, real-world examples, and step-by-step walkthroughs are available in the examples/notebook/ folder:

To launch the notebooks with all dependencies:

# Launch Jupyter Lab directly via uv
uv run --with jupyter jupyter lab examples/notebook/

How It Works

flowchart LR
    subgraph Client["Your Application / Pipeline"]
        A["RAG Context & Response"] --> C["LayaEval SDK"]
        B["Agent Execution Trace"] --> C
    end

    subgraph Batcher["LayaEval Orchestrator"]
        C --> D["Pack Questions & States"]
        D --> E["Single JSON-RPC / MCP Request"]
    end

    subgraph MCP["Laya MCP Server"]
        E --> F["Non-Autoregressive Neural Model"]
        F --> G["Multi-Question Forward Pass"]
    end

    G --> H["Calibrated Verdicts\nScore (0-4) • P(true) • Confidence"]
    H --> C

1. State Aggregation: You supply standard state keys (query, retrieved_context, response, agent_trace, etc.).
2. Batch Construction: LayaEval compiles the requested metrics into typed neural decision prompts.
3. Sub-50ms Inference: The Laya MCP Server runs a non-autoregressive forward pass, returning calibrated probabilities rather than generating arbitrary text tokens.
4. Normalized Results: Every output is mapped to a typed EvalResult with normalized scores, confidence intervals, and pass/fail statuses against your specified thresholds.

Supported Metrics Catalog

LayaEval uses three calibrated output types:

• Score (0.0 – 4.0): Graded rubric (0: poor, 1: weak, 2: partial, 3: mostly good, 4: perfect). Default pass threshold is $\ge 3.0$.
• Probability $P(\text{true})$ (0.0 – 1.0): Calibrated likelihood for positive qualities. Default pass threshold is $\ge 0.50$ (or $\ge 0.70$).
• Risk Probability $P(\text{true}) \le \text{threshold}$: Calibrated likelihood of negative events (hallucination, contradiction, security injection). Safe pass threshold is $\le 0.20$ or $\le 0.30$.

1. RAG Metrics Guide (20 Metrics)

📓 Interactive Notebook: Open examples/notebook/rag_metrics_evaluation.ipynb for runnable code examples of each RAG metric.

2. Agent Metrics Guide (22 Metrics)

📓 Interactive Notebook: Open examples/notebook/agent_metrics_evaluation.ipynb for runnable code examples of each Agent metric.

CLI Usage

LayaEval includes a command-line interface for testing, batch benchmarking, and regression comparison:

# List all 42 available metrics with descriptions
layaeval list-metrics

# Evaluate a JSONL dataset with selected metrics
layaeval run dataset.jsonl --metrics groundedness hallucination task_completed --output report.json

# Compare candidate run against a baseline to check for regressions
layaeval compare baseline_report.json candidate_report.json

Development & Testing

Run unit tests and lint checks using uv:

# Install dev dependencies
uv sync --all-extras

# Run test suite
uv run pytest

# Check code formatting and linting
uv run ruff check .

Acknowledgements & Special Thanks

Special thanks and gratitude to the Laya open-source contributors and community. LayaEval is powered by the foundational non-autoregressive neural architectures, latency breakthroughs, and high-performance Model Context Protocol (MCP) server engineered by the Laya project.

• Laya GitHub Repository: github.com/NandhaKishorM/laya
• Laya MCP Ecosystem & Docs: github.com/NandhaKishorM/laya/blob/main/docs

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.