lexigram-ai-evaluation

AI Evaluation framework for the Lexigram Framework.

Overview

AI Evaluation framework for the Lexigram Framework. Provides evaluators harness, and metrics — all wired through the DI container via EvaluationModule. Zero-config usage starts with sensible defaults.

Install

uv add lexigram-ai-evaluation
# Optional extras
uv add "lexigram-ai-evaluation[openai,anthropic]"

Quick Start

from lexigram import Application
from lexigram.di.module import Module, module

from lexigram.ai.evaluation import EvaluationModule
from lexigram.ai.evaluation.config import EvaluationConfig

@module(imports=[
    EvaluationModule.configure(
        EvaluationConfig(default_threshold=0.8)
    )
])
class AppModule(Module):
    pass

app = Application(modules=[AppModule])
if __name__ == "__main__":
    app.run()

Configuration

Zero-config usage: Call EvaluationModule.configure() with no arguments to use defaults.

Option 1 — YAML file

# application.yaml
ai_evaluation:
  enabled: true
  default_threshold: 0.8
  embedding_model: "text-embedding-3-small"

Option 2 — Profiles + Environment Variables (recommended)

export LEX_AI_EVALUATION__DEFAULT_THRESHOLD=0.8
# Environment variables for each field

Option 3 — Python

from lexigram.ai.evaluation.config import EvaluationConfig
from lexigram.ai.evaluation import EvaluationModule

config = EvaluationConfig(
    default_threshold=0.8,
    embedding_model="text-embedding-3-small",
)
EvaluationModule.configure(config)

Config reference

Module Factory Methods

Key Features

• 5 evaluator types: Criteria, QA, Embedding Distance, String Distance, Trajectory
• Evaluation harness: Run datasets against models with configurable evaluators
• Metrics: Pass/fail, score-based, and threshold evaluation
• Embedding support: Semantic similarity via embeddings

Testing

async with Application.boot(modules=[EvaluationModule.stub()]) as app:
    # your test code
    ...

Key Source Files