GLLM Evaluator SDK

A comprehensive evaluation framework for Generative AI applications including LLM outputs, AI Agent responses, and RAG (Retrieval-Augmented Generation) systems.

Overview

The GLLM Evaluator SDK provides a robust, extensible framework designed to make AI evaluation as simple and seamless as possible across the GDP Labs ecosystem. Built with integration-first philosophy, it enables teams to easily assess the quality of generated content from any AI system while seamlessly connecting with experiment tracking and observability platforms.

Philosophy

Easy Evaluation Everywhere: Standardize evaluation practices across all GDP Labs AI applications with minimal setup and maximum flexibility.

Integration-First Design: Built to work seamlessly with your existing experiment tracking, observability, and MLOps infrastructure.

Extensible by Design: Add new evaluators, metrics, and integrations without breaking existing workflows.

Key Features

• 🌐 GDP Labs Ecosystem Ready: Standardized evaluation framework across all internal AI applications
• 🔌 Seamless Integration: Easy integration with experiment tracking and observability platforms
• 🚀 Async-First Design: High-performance async evaluation with parallel processing
• 🔧 Extensible Architecture: Easy to add new evaluators and metrics for any use case
• 🤖 LLM as a Judge: Advanced language models for nuanced, contextual evaluation
• 📐 Traditional Metrics: Support for classical evaluation metrics and custom scoring functions
• 🔗 Popular Evaluator Integration: Integration with popular evaluators such as RAGAS, DeepEval, and LangChain
• ⚡ Zero-Config Start: Get started with sensible defaults, customize as needed

Installation

Prerequisites

Mandatory:

1. Python 3.11+ — Install here
2. pip — Install here
3. uv — Install here
4. gcloud CLI (for authentication) — Install here, then log in using:

   gcloud auth login
   

Install from Artifact

Because gllm-evals is a private library hosted in a secure Google Cloud repository, you must provide an access token to install it. The command below handles this authorization inline by using an access token from the gcloud CLI.

uv pip install --extra-index-url https://oauth2accesstoken:$(gcloud auth print-access-token)@glsdk.gdplabs.id/gen-ai-internal/simple/ gllm-evals

Local Development Setup

Prerequisites

1. Python 3.11+ — Install here

2. pip — Install here

3. uv — Install here

4. gcloud CLI — Install here, then log in using:

   gcloud auth login
   

5. Git — Install here

6. Trivy — Install here (required by the pre-push git hook, which runs a vulnerability scan on changed libraries before every push)

7. Access to the GDP Labs SDK GitHub repository

1. Clone Repository

git clone git@github.com:GDP-ADMIN/gl-sdk.git
cd gl-sdk/libs/gllm-evals

2. Setup Authentication

Because gllm-evals is a private library, you first need to configure uv to authenticate with our secure Google Cloud repositories. Set the following environment variables to authenticate with internal package indexes:

export UV_INDEX_GEN_AI_INTERNAL_USERNAME=oauth2accesstoken
export UV_INDEX_GEN_AI_INTERNAL_PASSWORD="$(gcloud auth print-access-token)"

3. Quick Setup

Run:

make setup

4. Activate Virtual Environment

source .venv/bin/activate

Local Development Utilities

The following Makefile commands are available for quick operations:

Install uv

make install-uv

Install Pre-Commit

make install-pre-commit

This installs both the pre-commit and pre-push git hooks from the repository-root .pre-commit-config.yaml. The pre-push hook runs unit tests, a changed-line coverage check, and a Trivy vulnerability scan for every changed library before a push completes, so make sure Trivy is installed and gcloud auth login is active.

Install Dependencies

make install

Update Dependencies

make update

Run Tests

make test

Adding the Package

Once authorization is configured, you can add gllm-evals to your project:

uv add gllm-evals

Dependencies

The SDK requires:

• gllm-core and gllm-inference for LLM interactions
• pydantic for data validation

Quick Start

Basic Usage

import asyncio
import os
from gllm_evals.evaluator.geval_generation_evaluator import GEvalGenerationEvaluator

async def main():
    # Initialize the evaluator
    evaluator = GEvalGenerationEvaluator(
        model_credentials=os.getenv("GOOGLE_API_KEY")
    )

    # Prepare evaluation data
    data = {
        "query": "What is the capital of France?",
        "expected_response": "Paris is the capital of France.",
        "generated_response": "The capital of France is Paris.",
        "retrieved_context": "Paris is the capital and largest city of France."
    }

    # Evaluate
    result = await evaluator.evaluate(data)
    print(result)

if __name__ == "__main__":
    asyncio.run(main())

Multimodal Evaluation

Use [ATTACHMENT:<id-or-uri>] placeholders in evaluation fields when the judge needs image context. Attachments may be declared by ID through LLMTestCase.attachments, or referenced inline with https://, http://, file://, or data: URIs.

import asyncio

from gllm_evals import AttachmentRef, LLMTestCase, evaluate
from gllm_evals.evaluator.geval_generation_evaluator import GEvalGenerationEvaluator


async def main():
    data = [
        LLMTestCase(
            input="Describe this image: [ATTACHMENT:sample_image]",
            actual_output="The image shows a mountain landscape.",
            expected_output="A mountainous outdoor landscape is visible.",
            retrieved_context=["Reference image: [ATTACHMENT:sample_image]"],
            attachments={
                "sample_image": AttachmentRef(
                    uri="https://picsum.photos/id/29/640/480.jpg",
                    mime_type="image/jpeg",
                )
            },
        )
    ]

    results = await evaluate(data=data, evaluators=[GEvalGenerationEvaluator()])
    print(results)


if __name__ == "__main__":
    asyncio.run(main())

You can place image placeholders in input, actual_output, expected_output, retrieved_context, and expected_context. The evaluator resolves placeholders before invoking the judge; unresolved placeholders fail validation instead of being sent as raw text.

For CSV and spreadsheet datasets, put row-level attachments in an attachments JSON cell:

input,actual_output,expected_output,attachments
"Describe [ATTACHMENT:image_1]","A mountain landscape.","A mountain landscape.","{""image_1"":{""uri"":""https://picsum.photos/id/29/640/480.jpg"",""mime_type"":""image/jpeg""}}"

See examples/evaluate/example_multimodal_evaluate_from_csv.py for a runnable CSV example that references bundled local image files, including a chart image used for a chart-comprehension question.

Inline URI and data URL placeholders are also supported without a named attachment entry:

LLMTestCase(
    input="Describe [ATTACHMENT:https://example.com/image.jpg]",
    actual_output="A product photo.",
)

Pass multimodal=False to suppress multimodal judging rules on the metrics that expose the option — LMBasedMetric and the GEval-family metrics (DeepEvalGEvalMetric and its subclasses, e.g. groundedness, completeness, and redundancy). Placeholder resolution still occurs, so raw [ATTACHMENT:...] text is not sent to the judge.

Batch Evaluation

import asyncio
import os
from gllm_evals.dataset.dict_dataset import DictDataset
from gllm_evals.evaluator.geval_generation_evaluator import GEvalGenerationEvaluator
from gllm_evals.runner import Runner
from gllm_evals.experiment_tracker.csv_experiment_tracker import CSVExperimentTracker

async def batch_evaluation():
    # Initialize evaluator
    evaluator = GEvalGenerationEvaluator(
        model_credentials=os.getenv("GOOGLE_API_KEY"),
        run_parallel=True  # Enable parallel processing
    )

    # Create dataset
    dataset = DictDataset([
        {
            "query": "What is the capital of France?",
            "expected_response": "Paris",
            "generated_response": "Paris is the capital of France.",
            "retrieved_context": "Paris is the capital of France."
        },
        {
            "query": "What is 1 + 1?",
            "expected_response": "2",
            "generated_response": "The answer is 2.",
            "retrieved_context": "1 + 1 equals 2."
        }
    ])

    # Run evaluation
    runner = Runner(evaluator, batch_size=10)
    results = await runner.evaluate(dataset)

    # Track results
    tracker = CSVExperimentTracker(score_key="generation/score")
    tracker.log_batch(results)

    print(f"Evaluation Results: {tracker.get_results()}")

if __name__ == "__main__":
    asyncio.run(batch_evaluation())

Using Jev with GEval

Every GEval metric (built-in and custom DeepEvalGEvalMetric) accepts a decisions-model (DM) invoker such as TypeSafe Jev in models and fallback_models, next to regular LM invokers. Set OPENROUTER_API_KEY first.

import os

from gllm_inference.dm_invoker import build_dm_invoker
from gllm_inference.lm_invoker import build_lm_invoker
from gllm_evals.metrics.generation.geval_completeness import GEvalCompletenessMetric

jev = build_dm_invoker(
    model_id="openrouter/typesafe/jev-1.14",
    credentials=os.getenv("OPENROUTER_API_KEY"),
    config={"provider": {"order": ["typesafe"], "allow_fallbacks": False}},
)
deepseek = build_lm_invoker(
    model_id="openrouter/deepseek/deepseek-v4.1-flash-1017",
    credentials=os.getenv("OPENROUTER_API_KEY"),
    config={"provider": {"order": ["inference-net"], "allow_fallbacks": False}},
)

metric = GEvalCompletenessMetric(models=jev)                                         # Jev only
metric = GEvalCompletenessMetric(models=jev, fallback_models=[deepseek])             # Jev, then an LM
metric = GEvalCompletenessMetric(models=[jev, jev, jev])                             # three judges, majority vote

result = await metric.evaluate(row)
result.score                    # normalized score in [0, 1]
result.decision.choice          # e.g. "3", the rubric score Jev chose
result.decision.confidence      # e.g. 0.99
result.decision.probabilities   # e.g. {"1": 0.0, "2": 0.01, "3": 0.99}

• Fallback is per complete attempt, in the configured order. A judge that raises an invoker error, times out, or gives an unusable answer (a choice that was not offered) hands the row to the next judge. If exactly one judge was tried and failed, its own error is raised; if two or more failed, EvalJudgesExhaustedError is raised with every attempt in children. A configuration, input or unsupported-operation failure stops the chain at once.

• result.decision is set only when a single Jev judge produced the score. With several judges it is None on the aggregate; each judge's own entry keeps its decision.

• Fewshot, evaluation_steps, custom GEval and thresholds behave as with an LM judge. Jev receives only the configured steps, never steps another LM generated. For a criteria-only custom metric, Jev judges from the criteria and the rubric.

• YAML suites: give a model entry type: dm (type defaults to lm):

  evaluators:
    - class_name: GEvalGenerationEvaluator
      models:
        - type: dm
          model_id: openrouter/typesafe/jev-1.14
          credentials: ${OPENROUTER_API_KEY}
          config:
            provider:
              order: [typesafe]
              allow_fallbacks: false
  

• GEval only. Every other metric (LM-based, RAGAS, LangChain, the other DeepEval metrics) raises NotImplementedError naming the metric when handed a DM invoker, and so do the evaluators built on them (for example LMBasedRetrievalEvaluator).

• Fallback scope. The chain moves on when a judge's invoker fails or times out, when Jev's answer is unusable, and when an LM judge returns invalid JSON. Other errors, such as a row missing a required field, are raised.

• Rows with images. Jev cannot take images, so it is skipped for such a row and an image-capable LM in fallback_models judges it. If no configured judge can, the row fails with EvalUnsupportedOperationError. A capability probe that fails for an operational reason (authentication, rate limit, connection, timeout) raises that failure instead and is never read as "this judge cannot take images".

• Batch API. A chain that contains Jev never uses the provider Batch API; list input is evaluated row by row.

• Token usage is reported per model; no cost is calculated for Jev.

• get_full_prompt returns the Jev request as JSON when Jev is the primary judge.

Custom Metrics

Create domain-specific metrics easily:

from gllm_evals.metrics.metric import BaseMetric
from gllm_evals.types import LLMTestCase, MetricOutput

class DomainSpecificMetric(BaseMetric):
    """Custom metric for domain-specific evaluation."""

    name = "domain_accuracy"

    async def _evaluate(self, data: LLMTestCase) -> MetricOutput:
        # Your domain-specific evaluation logic
        score = self.calculate_domain_score(data)
        return {"score": score, "explanation": "Domain-specific reasoning"}

A list-capable _evaluate that has no batch mode for its configuration signals it by raising BatchNotSupportedError (from gllm_evals.exceptions); the rows are then evaluated one by one. Any other exception raised for a list, including a bare TypeError or NotImplementedError, is treated as a failure.

Error Handling

Every error the SDK raises on purpose is a gllm_core.exceptions.BaseError with a stable code. gllm-evals' own errors are BaseEvalError subclasses with codes GLLM-EVAL-*, and each also inherits the closest built-in exception (for example EvalConfigurationError is a ValueError), so existing except ValueError blocks keep working. Errors from gllm-inference (GLLM-INF-*) pass through unchanged.

Each error also has recovery guidance (retryable, and retry_after_seconds when the service reported one) and, once it has been recorded, a reference_id.

Handle errors by class or code, never by message text. Messages are written for people and may change. Every message says what happened and then what to do next, whether it is a class's default message or a more specific one.

from gllm_core.exceptions import BaseError
from gllm_evals import evaluate
from gllm_evals.exceptions import EvalConfigurationError, EvalInputError

try:
    result = await evaluate(data="data.csv", evaluators=[evaluator])
except EvalInputError as error:           # the caller can fix the data
    print(error.code, error.message)
except EvalConfigurationError as error:   # the caller can fix the setup
    print(error.code, error.message)
except BaseError as error:                # any other normalized failure
    if error.recovery.get("retryable"):
        ...  # retry later
    raise

A failed row does not raise. When an evaluator or metric fails on a row, the run continues and that result carries an error record (an ErrorInfo model) instead of a score, with code, message (safe to show), reference_id, error_type, origin, recovery, component, operation and children (the failures it summarizes). A failed evaluator's result is keyed by the evaluator's name, like a successful one. Use is_failed_result to tell a failure from a valid low score:

from gllm_evals.exceptions import is_failed_result

for row in result.results:
    for evaluator_result in row:
        for name, output in evaluator_result.items():
            if is_failed_result(output):
                print(name, output.error.code, output.error.reference_id)

Result fields, messages and log records never contain row content, prompts, file paths, URLs, credentials or the upstream error text. Handled failures are logged once on the gllm_evals.errors logger with the code in extra["error_code"]; the original exception stays available as __cause__ on a raised error. Direct Pydantic model construction (for example LLMTestCase(...)) still raises pydantic.ValidationError.

See examples/example_error_handling.py for a runnable walkthrough.

Architecture

Core Components

1. Evaluators

• BaseEvaluator: Abstract base class for all evaluators - extend for any evaluation scenario
• GEvalGenerationEvaluator: Production-ready GEval-backed evaluator for text generation quality with rule-based scoring

2. Metrics

• BaseMetric: Abstract base class for metrics - create custom metrics for any domain
• LMBasedMetric: Generic LM-powered metric evaluation with customizable prompts

3. Datasets

• BaseDataset: Abstract base class for datasets - support any data format
• DictDataset: Simple dictionary-based dataset implementation

4. Runner

• Runner: Runner class for batch evaluation

Metrics

Below is a list of metrics that are currently supported by the SDK.

Evaluators

Below is a list of evaluators that are currently supported by the SDK.

Datasets

Below is a list of datasets that are currently supported by the SDK.