evallm-qa 0.1.1

QA framework for evaluating LLM outputs based on user-defined metrics

evaLLM

evaLLM is a lightweight QA framework for evaluating LLM outputs with composable, schema-validated metrics.

It is designed for two workflows:

• batch evaluation through a CLI
• embeddable evaluation pipelines through a Python API

Quick links

• Documentation: https://psandhaas.github.io/evaLLM/
• Repository: https://github.com/psandhaas/evaLLM

Installation

Install from PyPI

pip install evallm-qa

Install optional extras

Use this if you want encoding-based metrics (for example count_encodings):

pip install "evallm-qa[hf-tokenizers]"

Install from source (secondary option)

Use this if you want to work from a local checkout:

git clone https://github.com/psandhaas/evaLLM.git
cd evaLLM
pip install .

Source install with extras:

pip install .[hf-tokenizers]

What evaLLM evaluates

Current built-in metrics include:

• CountsMetric: character, token, or model-encoding counts
• JsonFormatMetric: field-level JSON format validation against a JSON Schema

Results are emitted per input record as JSON objects, which makes them easy to pipe, persist, and analyze downstream.

Library structure

evaLLM follows a layered architecture to keep concerns separated:

• evallm.application
  • orchestration layer
  • DTOs (EvaluationRequest, MetricSpec, dataset specs)
  • request resolution and evaluator execution
• evallm.metrics
  • base metric interface (BaseMetric[T])
  • built-in metrics and registration catalog
  • metric composition (CompositeMetric)
• evallm.readers
  • dataset access layer (currently JSONL reader)
• evallm.cli
  • Typer-based command-line interface

Further details on architectural choices can be found in the documentation.

Composite metrics

evaLLM intentionally models multi-metric evaluation using the Composite design pattern.

Composite is a good fit because each metric is an independent computation over the same input text, and the framework must aggregate all metric outputs into one result object. This creates a tree-shaped execution model:

• a parent metric container evaluates children
• each child contributes a namespaced result
• multiple instances of the same metric class can coexist with different configs

This gives you two concrete benefits:

• as a user you can rely on the fact that calling evalute(text) does exactly that; your input is evaluated against however many metrics, with whichever configuration you chose, and
• as a developer you only need to implement evaluate(text), without having to consider how or when your metric is called.

Core concepts

Evaluation request

An evaluation is defined by:

• one dataset spec (dataset)
• one or more metric specs (metrics)

Metric spec

Each metric spec supports:

• name: registered metric identifier
• config (optional): runtime metric config
• result_key (optional): output field name override

Dataset support

Current support:

• JSONL datasets (JsonlDatasetSpec)

Each JSONL record must contain a text field and can optionally provide a stable record id field.

Extend BaseReader and BaseDatasetSpec for further formats.

Python API examples

1) Build and run an evaluation with helper presets

from evallm.application import (
    JsonlEvaluationBuilder,
    count_characters,
    count_tokens,
    json_format,
)

builder = JsonlEvaluationBuilder(
    "./data/dataset.jsonl",
    text_key="text",
    record_id_key="id",
)

builder.use(count_characters(result_key="chars"))
builder.use(count_tokens(segmentation="wordpunct", result_key="tokens_wordpunct"))
builder.use(
    json_format(
        {
            "type": "object",
            "properties": {
                "answer": {"type": "string"},
                "confidence": {"type": "number", "minimum": 0.0, "maximum": 1.0},
            },
            "required": ["answer"],
        },
        result_key="format_ok",
    )
)

for result in builder.run():
    print(result.model_dump())

2) Build a request explicitly

from evallm.application import EvaluationRequest

request = EvaluationRequest.create(
    obj={
        "dataset": {
            "input_file": "./data/dataset.jsonl",
            "text_key": "text",
            "record_id_key": "id",
        },
        "metrics": [
            {"name": "CountsMetric", "result_key": "chars"},
            {
                "name": "CountsMetric",
                "config": {"segments": "tokens", "segmentation": "whitespace"},
                "result_key": "tokens",
            },
        ],
    }
)

CLI examples

Evaluate via flags

evallm run \
    --input-file ./data/dataset.jsonl \
    --text-key text \
    --record-id-key id \
    --metric CountsMetric

Evaluate via config file

evallm run --config ./run.yaml

Discover available metrics and presets

evallm metrics list
evallm metrics info CountsMetric --format yaml
evallm metrics presets

Write JSONL results to a file

evallm run \
    -f ./data/dataset.jsonl \
    -t text \
    -i id \
    -m CountsMetric \
    --output ./results.jsonl

Config file reference

evallm run expects two top-level sections:

1. dataset
2. metrics

Minimal YAML example:

dataset:
    input_file: ./data/dataset.jsonl
    text_key: text
    record_id_key: id

metrics:
    - name: CountsMetric
        result_key: chars

    - name: CountsMetric
        config:
            segments: tokens
            segmentation: whitespace
        result_key: tokens

    - name: JsonFormatMetric
        config:
            expected_schema:
                type: object
                properties:
                    answer:
                        type: string
                    confidence:
                        type: number
                        minimum: 0.0
                        maximum: 1.0
                required:
                    - answer
            check_formats: true
        result_key: structured_output

Important merge rule:

• if --config and inline flags are both provided, inline dataset flags override config fields
• if any --metric flags are provided, they replace config.metrics

Extending evaLLM with custom metrics

Implement BaseMetric[T] and register with @register_metric.

from evallm.metrics.base import BaseMetric
from evallm.metrics.registry import register_metric


@register_metric(name="token_lengths")
class TokenLengthsMetric(BaseMetric[list[int]]):
    def evaluate(self, text: str) -> dict[str, list[int]]:
        if text.strip() == "":
            return self.result([])
        return self.result([len(tok) for tok in text.split()])

Development

Run tests:

uv run pytest

Without uv:

python -m pytest

Run docs locally:

uv run mkdocs serve

Without uv:

mkdocs serve

If you want to use uv, installation instructions are available at https://docs.astral.sh/uv/getting-started/installation/.