Gaussia

AI evaluation framework for measuring fairness, quality, and safety of AI models and assistants.

Installation

pip install gaussia

With specific metric dependencies:

pip install gaussia[toxicity]    # Toxicity analysis
pip install gaussia[bias]        # Bias detection
pip install gaussia[evalhub]     # EvalHub provider adapter
pip install gaussia[metrics]     # All metrics
pip install gaussia[all]         # Everything

Quick Start

from gaussia import Retriever, Dataset, Batch
from gaussia.metrics import Context

# 1. Define your data source
class MyRetriever(Retriever):
    def load_dataset(self) -> list[Dataset]:
        return [
            Dataset(
                session_id="session-1",
                assistant_id="assistant-1",
                language="en",
                context="France is a country in Western Europe.",
                conversation=[
                    Batch(
                        qa_id="q1",
                        query="Where is France?",
                        assistant="France is located in Western Europe.",
                        ground_truth_assistant="France is a country in Western Europe.",
                    )
                ],
            )
        ]

# 2. Run a metric
metrics = Context.run(retriever=MyRetriever())

Metrics

Features

Guardians

Pluggable bias detection backends:

from gaussia.guardians import IBMGraniteGuardian, LLamaGuardGuardian

metrics = Bias.run(retriever=MyRetriever(), guardian=IBMGraniteGuardian())

Statistical Modes

Choose between frequentist and Bayesian aggregation:

from gaussia import FrequentistMode, BayesianMode

metrics = Context.run(retriever=MyRetriever(), statistical_mode=FrequentistMode())
metrics = Context.run(retriever=MyRetriever(), statistical_mode=BayesianMode())

Synthetic Data Generation

Generate evaluation datasets from documents:

from gaussia.generators import BaseGenerator, create_markdown_loader

loader = create_markdown_loader(path="./docs")
generator = BaseGenerator(context_loader=loader)
datasets = generator.generate()

Explainability

Token-level attribution analysis:

from gaussia.explainability import AttributionExplainer

explainer = AttributionExplainer(method="lime")
attributions = explainer.explain(text="Your input text")

Prompt Optimization

Optimize prompts using evolutionary and multi-objective strategies:

from gaussia.prompt_optimizer import GEPAOptimizer, MIPROv2Optimizer

EvalHub Provider

Run Gaussia as an EvalHub BYOF provider:

python -m gaussia.integrations.evalhub.adapter

Documentation

Full documentation available at docs.gaussia.ai.

Requirements

• Python >= 3.11

License

MIT