ignis_evaluation

A config-driven LLM evaluation framework built on top of DeepEval.
Define metrics in a YAML bundle config, run evaluations via REST API or decorator, persist scores to PostgreSQL, and view them on a hierarchical evaluation dashboard.

---

Table of Contents

• Features
• Architecture
• Complete User Guide (MVP2)

---

Features

• YAML bundle configs — group metrics into named bundles; switch bundles per API call with no code change
• Hierarchical dashboard — domain → project → bundles → metrics with KPIs, 6 chart types, and status badges
• Observability integration — eval worker reads production traces from an Observability DB and evaluates them automatically
• Parallel metric execution — all metrics run concurrently via ThreadPoolExecutor
• PostgreSQL persistence — every run, test case, and metric score stored automatically
• @ignis_eval decorator — wrap any LLM function in your app to evaluate output inline
• REST API — trigger evaluations and retrieve results over HTTP
• Confident AI upload — optional push to Confident AI cloud dashboard
• Offline mode — run without an OpenAI key using use_deepeval: false

---

Architecture

Evaluation Flow Pattern

                          ignis_evaluation — Evaluation Flow

 ┌──────────────────────────────┐
 │         Your App             │
 │                              │
 │  Option A: REST API call     │
 │  POST /evaluate              │
 │                              │
 │  Option B: @ignis_eval       │
 │  decorator on any function   │
 └──────────────┬───────────────┘
                │
                ▼
 ┌──────────────────────────────┐
 │       EvaluationRunner       │
 │                              │
 │  1. Loads YAML bundle config │
 │  2. Resolves metric classes  │
 │  3. Builds DeepEval objects  │
 └──────────────┬───────────────┘
                │
                ▼
 ┌──────────────────────────────┐       ┌─────────────────────────────┐
 │   ThreadPoolExecutor         │──────▶│   OpenAI / LLM Judge        │
 │                              │       │   (gpt-4o-mini or gpt-4o)   │
 │  All metrics run in parallel │◀──────│   Scores each metric        │
 │  AnswerRelevancy · GEval     │ scores│   Returns score + reason    │
 │  Faithfulness · Hallucination│       └─────────────────────────────┘
 └──────────────┬───────────────┘
                │  weighted score + per-metric results
                ▼
 ┌──────────────────────────────┐
 │    EvaluationRunner          │
 │    (collects & aggregates)   │
 └──────┬───────────────┬───────┘
        │               │
        ▼               ▼
 ┌─────────────┐  ┌─────────────────┐
 │  PostgreSQL │  │  JSON + CSV     │
 │  eval runs  │  │  reports/       │
 │  test cases │  │  (local files)  │
 │  metrics    │  └─────────────────┘
 └─────────────┘

Observability Integration Pattern

                          Observability → Evaluation Pipeline

 ┌─────────────────────┐     writes     ┌──────────────────────┐
 │   Production App    │───────────────▶│   Observability DB   │
 │   (Your LLM)        │    traces      │   traces table       │
 └─────────────────────┘                └──────────┬───────────┘
                                                    │
                                          polls for new traces
                                          (ingested_at > last checkpoint)
                                                    │
                                                    ▼
                                         ┌──────────────────────┐
                                         │     Eval Worker      │
                                         │   eval_worker.py     │
                                         │                      │
                                         │  Reads new traces    │
                                         │  Saves checkpoint    │
                                         │  Batches to API      │
                                         └──────────┬───────────┘
                                                    │
                                          POST /evaluate/batch
                                                    │
                                                    ▼
                                         ┌──────────────────────┐
                                         │   ignis_evaluation   │
                                         │   REST API           │
                                         │   (EvaluationRunner) │
                                         └──────────┬───────────┘
                                                    │
                                  ┌─────────────────┴──────────────────┐
                                  ▼                                     ▼
                       ┌──────────────────┐               ┌────────────────────────┐
                       │   PostgreSQL     │               │   Confident AI         │
                       │   Evaluation DB  │               │   (Cloud Dashboard)    │
                       │   metric_results │               │   optional upload      │
                       │   trace_id link  │               └────────────────────────┘
                       └──────────────────┘

---

Complete User Guide (MVP2)

A config-driven LLM evaluation framework built on DeepEval.
Evaluate LLM outputs via REST API, Python decorator, or automated observability worker.
Persist results to PostgreSQL and visualise them on a hierarchical dashboard.

---

Table of Contents

• Prerequisites
• Installation
• Configuration
• Database Setup
• Starting the API Server
• Project Structure
• Built-in Bundles
• Core Usage
  • Single Evaluation
  • Batch Evaluation
  • Conversational Evaluation
  • Python Decorator
  • Custom Bundles
• Dashboard API
• Observability Worker
• Custom Metrics
• Troubleshooting

---

Prerequisites

Requirement	Version	Notes
Python	3.10 or higher	Required for match statements and modern typing
PostgreSQL	14+	For result persistence. Optional — runs without it
OpenAI API Key	—	Required for real LLM-judged evaluations
pip	latest	python -m pip install --upgrade pip

---

Installation

Option A — Install from PyPI (recommended for integration)

pip install ignis_evaluation

Option B — Install from source (for development or contribution)

git clone https://github.com/Infogain-GenAI/ignis_evaluation.git
cd ignis_evaluation

python -m venv venv

# Windows
venv\Scripts\activate

# macOS / Linux
source venv/bin/activate

pip install -e .          # core install
pip install -e ".[api]"   # adds FastAPI + Uvicorn for the REST server
pip install -e ".[dev]"   # adds pytest and test tools

---

Configuration

Copy the example environment file and fill in your values:

# Windows
copy .env.example .env

# macOS / Linux
cp .env.example .env

Minimum required

OPENAI_API_KEY=sk-proj-your-key-here
# EVAL_DB_URL=postgresql://postgres:password@localhost:5432/deepeval_results

Full reference

Variable	Default	Purpose
OPENAI_API_KEY	—	Required for live LLM-judged evaluations
IGNIS_MODEL	gpt-4o-mini	LLM judge model for all evaluations. Always set this explicitly — if unset, the base config file's model: field takes effect, which may be a slower/more expensive model
EVAL_DB_URL	—	PostgreSQL connection string for result storage
EVAL_REPORTS_DIR	./reports	Where JSON/CSV report files are saved
PROJECT_DOMAIN	—	Domain tag applied when a new project is created (e.g. Finance)
IGNIS_DEFAULT_PROJECT	—	Default project name for single-app deployments
IGNIS_DEFAULT_APPLICATION	—	Default application label (sprint, version, campaign)
IGNIS_BUNDLE_DIR	—	Path to your custom bundle YAML folder
IGNIS_BUNDLE_STRICT	false	true = refuse startup if any custom bundle YAML is invalid
CORS_ALLOWED_ORIGINS	—	Comma-separated origins allowed for browser clients
OBS_DB_URL	—	Observability DB (read-only) — required only for the eval worker
EVAL_API_URL	http://localhost:8001	API URL used by the eval worker
EVAL_WORKER_BATCH_SIZE	100	Max traces per worker run
EVAL_WORKER_LOOKBACK_DAYS	7	How far back to look when no checkpoint exists
CONFIDENT_API_KEY	—	Optional — upload results to Confident AI cloud
DEEPEVAL_TELEMETRY_OPT_OUT	—	Set YES to disable DeepEval usage telemetry

---

Database Setup

Skip this section if you want to run without persistence — evaluations still work and results are saved to JSON/CSV reports.

Step 1 — Create the database

-- Run in pgAdmin or psql
CREATE DATABASE deepeval_results;

Step 2 — Apply the schema

psql -U postgres -d deepeval_results -f src/ignis_evaluation/sql/schema.sql

This creates:

• projects — project registry (UUID primary key)
• evaluation_runs — one row per evaluation call
• test_results — one row per test case
• metric_results — one row per metric per test case
• bundle_registry — auto-populated with built-in bundles on first startup
• eval_worker_checkpoints — observability worker audit trail
• v_run_metrics — flat view joining all tables for dashboards

---

Starting the API Server

uvicorn src.ignis_evaluation.api.main:app --reload --port 8001

Interactive docs (Swagger UI): http://localhost:8001/docs

The service auto-registers all built-in bundles into the database on startup.

Important — .env changes require a full server restart.
--reload only watches Python source files. If you change any value in .env (model, API key, DB URL, etc.), you must stop the server (Ctrl+C) and start it again. The new env values will not be picked up by a hot reload.

---

Expected Evaluation Timing

Evaluation time depends on the bundle and model used. All metrics in a bundle run concurrently.

Bundle	Metrics	Expected time (gpt-4o-mini)
production_monitor	1 metric	~5–8s
core_quality	3 metrics	~20–35s
customer_support	3 metrics	~20–35s
safety_guard	2 metrics	~10–20s
rag_standard	6 metrics	~30–50s
full_audit	8+ metrics	~40–70s

Times vary with OpenAI network latency. Each metric makes 2–3 internal LLM calls sequentially; the bundle total is bounded by the slowest metric, not their sum.

Using gpt-4o instead of gpt-4o-mini increases times by 3–5×. Always set IGNIS_MODEL=gpt-4o-mini in .env unless you have a specific reason to use a larger model.

---

Project Structure

ignis_evaluation/
├── api/
│   ├── main.py                     # FastAPI entry point
│   ├── models.py                   # Pydantic request/response models
│   └── routers/
│       ├── evaluate.py             # POST /evaluate, /evaluate/batch, /evaluate/conversational
│       ├── bundles.py              # GET /bundles, GET /bundles/{id}
│       ├── analytics.py            # POST /eval/dashboard + KPI/chart endpoints
│       ├── projects.py             # GET /projects, GET /projects/{name}/applications
│       ├── results.py              # GET /results, GET /results/{run_id}
│       ├── health.py               # GET /health
│       └── metrics.py              # GET /metrics
├── bundle_registry/                # Built-in bundle YAML files + loader
│   ├── core_quality.yaml
│   ├── rag_standard.yaml           # All 6 RAG metrics
│   ├── safety_guard.yaml
│   ├── summarization.yaml
│   ├── production_monitor.yaml
│   ├── customer_support.yaml
│   ├── compliance_finance.yaml
│   ├── chatbot_quality.yaml        # Conversational metrics
│   └── validator.py                # Bundle YAML validation
├── db/
│   ├── connection.py
│   ├── writer.py                   # INSERT evaluation runs
│   ├── reader.py                   # SELECT runs for Results API
│   └── analytics.py                # Dashboard queries
├── eval_library/
│   ├── runner.py                   # Core evaluation engine
│   ├── decorator.py                # @ignis_eval decorator
│   ├── conversational_runner.py    # Multi-turn conversation evaluation
│   ├── metric_registry.py          # Maps metric names → DeepEval classes
│   └── scorer.py                   # Weighted composite scoring
├── services/
│   └── dashboard_service.py        # build_enriched_dashboard()
├── sql/
│   └── schema.sql                  # Database schema (run once)
└── workers/
    └── eval_worker.py              # Observability → Eval pipeline

---

Built-in Bundles

Eight bundles ship with the package. Pass the bundle name in any API call.

Bundle ID	Metrics	Use Case	Context Required?
core_quality	AnswerRelevancy, Clarity, Conciseness	Default for any chat app	No
rag_standard	All 6 RAG metrics (Faithfulness, AnswerRelevancy, Hallucination, ContextualRelevancy, ContextualPrecision, ContextualRecall)	RAG pipelines	Yes
safety_guard	Toxicity, Bias (GEval)	Public-facing deployments	No
summarization	Summarization, Faithfulness	Document summarization	Yes
production_monitor	AnswerRelevancy only	Lightweight live traffic monitoring	No
customer_support	AnswerRelevancy, Helpfulness, Empathy	Support chatbots	No
compliance_finance	FinanceCompliance, AnswerRelevancy	Finance domain	No
chatbot_quality	ConversationCompleteness, RoleAdherence, KnowledgeRetention	Multi-turn chatbot sessions	Use with /evaluate/conversational

List all available bundles at any time:

GET http://localhost:8001/bundles

---

Core Usage

Prerequisite: Start the API server before running any curl example below:

uvicorn ignis_evaluation.api.main:app --reload --port 8001

(See Starting the API Server for the full command when running from source.)

1. Single Evaluation

Evaluate one input/output pair.

Reports: Add "save_report": true to the request body to also write a JSON+CSV file to EVAL_REPORTS_DIR (prefixed api_...). Omit it and no report file is written — only the JSON response is returned.

curl -X POST http://localhost:8001/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "project":     "CustomerSupportBot",
    "application": "sprint-23",
    "bundle":      "core_quality",
    "input":       "How do I reset my password?",
    "output":      "Click Forgot Password on the login screen and follow the link sent to your email."
  }'

With RAG context:

curl -X POST http://localhost:8001/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "project":     "MedicalRAG",
    "application": "v3",
    "bundle":      "rag_standard",
    "input":       "What is the dosage for ibuprofen?",
    "output":      "Standard adult dosage is 200-400mg every 4-6 hours.",
    "context":     ["Ibuprofen is an NSAID. Adult dose 200-400mg every 4-6h."],
    "expected_output": "200-400mg every 4-6 hours for adults."
  }'

Available fields:

Field	Required	Description
project	Recommended	Project name — stored in DB and dashboard
application	Recommended	Version/sprint/campaign label
bundle	Yes (or config)	Bundle ID from the registry
input	Yes	User query
output	Yes	LLM answer to evaluate
context	For RAG bundles	Retrieved chunks
expected_output	Optional	Ground truth — improves ContextualPrecision/Recall
model	Optional	Override LLM judge for this call only

---

2. Batch Evaluation

Evaluate multiple cases in one request. Useful for observability trace batches.

curl -X POST http://localhost:8001/evaluate/batch \
  -H "Content-Type: application/json" \
  -d '{
    "project":     "CustomerSupportBot",
    "application": "sprint-23",
    "bundle":      "core_quality",
    "cases": [
      {"input": "What is ML?",  "output": "ML learns patterns from data.", "test_name": "trace-001"},
      {"input": "What is RAG?", "output": "RAG retrieves documents before generating.", "test_name": "trace-002"}
    ]
  }'

Each case can independently have context, expected_output, and test_name.
Pass the observability trace_id as test_name to link eval scores back to production traces.

---

3. Conversational Evaluation

Evaluate a full multi-turn conversation session.

curl -X POST http://localhost:8001/evaluate/conversational \
  -H "Content-Type: application/json" \
  -d '{
    "project":     "CustomerSupportBot",
    "application": "sprint-23",
    "bundle":      "chatbot_quality",
    "turns": [
      {"user": "Hi, I cannot log in.", "assistant": "Can you tell me the error message?"},
      {"user": "It says password incorrect.", "assistant": "I will send a reset link to your email."},
      {"user": "Got it, reset done.", "assistant": "Great! You should be able to log in now."}
    ],
    "test_name": "session-abc123"
  }'

Conversational metrics evaluate the full conversation as a unit, not individual turns:

• ConversationCompletenessMetric — did the bot fully resolve the user's need?
• RoleAdherenceMetric — did the bot stay in its assigned persona/role?
• KnowledgeRetentionMetric — did the bot remember what was said in earlier turns?

---

4. Python Decorator

Wrap any LLM function to evaluate its output inline.

Installation (from your app):

pip install ignis_evaluation

Bundle-only (recommended — no config file needed):

from dotenv import load_dotenv
load_dotenv(".env")

import openai, openai.resources.chat  # pre-import to avoid thread deadlock
from ignis_evaluation.eval_library.decorator import ignis_eval

@ignis_eval(
    bundle="core_quality",
    project="CustomerSupportBot",
    application="sprint-23",
    include_results=True,
)
def handle_query(*, input_text: str, output_text: str):
    return {"input_text": input_text, "output_text": output_text}

result = handle_query(
    input_text="How do I reset my password?",
    output_text="Click Forgot Password on the login screen.",
)

print("Score:", result["evaluation"]["weighted_score"])
print("Bundle:", result["evaluation"]["bundle"])
for metric, data in result["evaluation"]["metrics"].items():
    status = "PASS" if data.get("pass") else "FAIL"
    print(f"  {metric}: {data.get('score')} — {status}")

With full report in result:

@ignis_eval(
    bundle="rag_standard",
    project="MedicalRAG",
    application="v3",
    include_results=True,
    include_reports=True,   # adds "report" key with serializable report dict
    save_report=True,       # also saves JSON+CSV to EVAL_REPORTS_DIR
)
def rag_answer(*, input_text, output_text, context):
    return {"input_text": input_text, "output_text": output_text, "context": context}

Decorator parameters:

Parameter	Default	Description
bundle	—	Bundle ID from registry (recommended)
config	—	YAML config name (V1 compat)
project	env default	Project name
application	env default	Application label
include_results	False	Return {"result": ..., "evaluation": {...}}
include_reports	False	Also include "report" key in return dict
save_report	False	Save JSON+CSV files to EVAL_REPORTS_DIR
eval_model	IGNIS_MODEL	Override LLM judge for this decorator

Note — reports are opt-in here too: Just like the API's save_report field, the decorator's save_report defaults to False. Set save_report=True explicitly to get JSON+CSV files (prefixed decorator_...) written to EVAL_REPORTS_DIR — otherwise only DB persistence (if configured) happens, no report files.

---

5. Custom Bundles

Create your own bundle YAML outside the package.

Step 1 — Create your bundles folder and YAML file:

your_project/
├── my_bundles/
│   └── support_quality.yaml
└── .env

# ./my_bundles/support_quality.yaml
bundle_id:   support_quality
description: "Quality check for customer support responses."

metrics:
  - name:      AnswerRelevancyMetric
    threshold: 0.70
    weight:    40

  - name:      GEvalMetric
    geval_name: "Clarity"
    criteria:   "The response must be clear and easy to understand for a non-expert."
    evaluation_params: [input, actual_output]
    threshold:  0.65
    weight:     35

  - name:      GEvalMetric
    geval_name: "Politeness"
    criteria:   "The response must be polite and professional in tone."
    evaluation_params: [input, actual_output]
    threshold:  0.60
    weight:     25

Weights across all metrics in a bundle must sum to exactly 100. Add as many YAML files as you want — one bundle per file.

Step 2 — Point the service to your folder in .env:

IGNIS_BUNDLE_DIR=./my_bundles
IGNIS_BUNDLE_STRICT=false

Step 3 — Restart the server, then use by name exactly like built-in bundles:

uvicorn ignis_evaluation.api.main:app --reload --port 8001

POST /evaluate
{
  "project": "CustomerSupportBot",
  "bundle":  "support_quality",
  "input":   "My order hasn't arrived, what do I do?",
  "output":  "I'm sorry for the delay! I've checked your order and it's on the way — you should receive it within 2 days."
}

Validation rules: The service validates every YAML at startup.

Field	Rule
bundle_id	Required, non-empty string
metrics	Required, non-empty list
name (per metric)	Must be a known built-in metric name or include module:
threshold	Float 0.0–1.0
weight	Positive number
criteria	Required when name is GEvalMetric

Set IGNIS_BUNDLE_STRICT=true to refuse service startup on invalid bundles (recommended for production).

---

Dashboard API

Single endpoint for KPIs, charts, and the full project hierarchy.

POST http://localhost:8001/eval/dashboard

Filter by project:

{
  "project":     "CustomerSupportBot",
  "window_hours": 168
}

Filter by domain:

{
  "domain":      "Finance",
  "window_hours": 168
}

Include recent evaluations:

{
  "project":             "CustomerSupportBot",
  "window_hours":         168,
  "include_evaluations":  true,
  "eval_limit":           20
}

Response structure:

{
  "kpis":     { evaluation_count, avg_weighted_score, pass_rate_pct, total_cost_usd, ... },
  "charts":   { score_over_time, pass_fail, cost_by_metric, score_by_metric, ... },
  "projects": [
    {
      "project_name":  "CustomerSupportBot",
      "domain":        "Finance",              ← metadata on project
      "applications": [
        {
          "application_name": "sprint-23",
          "bundles": [
            {
              "bundle":           "core_quality",
              "avg_weighted_score": 0.87,
              "metrics": [ ... ]
            }
          ]
        }
      ]
    }
  ]
}

Other useful endpoints:

Endpoint	Description
GET /bundles	List all available bundle IDs
GET /bundles/{bundle_id}	Full bundle detail — metrics, thresholds, weights
GET /projects	List all projects
GET /projects/{name}/applications	List applications under a project with stats
GET /results	Paginated list of evaluation runs
GET /results/{run_id}	Full detail for one run with all metric scores
GET /eval/metrics/leaderboard	Per-metric pass rate ranking
GET /eval/projects/{name}/trend	Score trend over time for a project
GET /eval/domains	List all domains with project and run counts
GET /health	Service health check

---

Observability Worker

Automatically evaluates production LLM traces from an observability database.

How it works:

1. Reads new traces from your Observability DB (traces table with input, output, trace_id)
2. Sends them as a batch to POST /evaluate/batch
3. Links each eval score back to its trace via test_name = trace_id
4. Saves a checkpoint so next run only processes new traces

Configure in .env:

OBS_DB_URL=postgresql://user:pass@host:5432/your_obs_db
EVAL_API_URL=http://localhost:8001
EVAL_WORKER_BATCH_SIZE=100
EVAL_WORKER_LOOKBACK_DAYS=7

# Map obs project names → bundle (JSON string)
EVAL_PROJECT_CONFIG_MAP={"my_chatbot": {"config": "chatapp", "bundle": "core_quality"}}

Run the worker:

# All projects
python -m ignis_evaluation.workers.eval_worker

# Single project
python -m ignis_evaluation.workers.eval_worker --project my_chatbot

# With bundle override
python -m ignis_evaluation.workers.eval_worker --project my_chatbot --bundle rag_standard

# Dry run (logs without calling the API)
python -m ignis_evaluation.workers.eval_worker --dry-run

Worker runs appear in the dashboard under application: "observability" separate from direct API calls.

---

Custom Metrics

Three ways to add a custom metric — start with the simplest that fits your need.

Tier	Who	What	Python file?
1 — KeywordCheckMetric	Anyone	Phrase / keyword checks via YAML lists	No
2 — GEvalMetric + criteria	Anyone	Plain-English LLM-judged rule	No
3 — Custom Python class	Developers	Any logic — regex, ML model, API call	Yes

---

Tier 1 — KeywordCheckMetric (no Python, YAML only)

The simplest option. Add it to any bundle YAML with two lists — no Python file, no class, no scoring formula.

# my_bundles/brand_quality.yaml
bundle_id: brand_quality

metrics:
  - name:      AnswerRelevancyMetric
    threshold: 0.70
    weight:    50

  - name:             KeywordCheckMetric
    must_contain:     ["happy to help", "let me", "feel free"]
    must_not_contain: ["obviously", "you must", "that's wrong"]
    scoring:          fraction       # fraction = partial credit | all_required = all or nothing
    threshold:        0.50
    weight:           50

Fields:

Restart required after adding a new bundle YAML.
Bundle YAMLs are loaded once at server startup. Add your YAML to IGNIS_BUNDLE_DIR and restart uvicorn before using the new bundle.

Field	Required	Description
must_contain	One of the two is required	Phrases that should appear in the output
must_not_contain	One of the two is required	Phrases that must NOT appear
scoring	No (default: fraction)	fraction = partial credit · all_required = all or nothing
threshold	Yes	Pass/fail threshold (0.0–1.0)
weight	Yes	Bundle weight (all weights must sum to 100)

Scoring rules:

1. If any must_not_contain phrase is found → score = 0.0 (hard fail, overrides everything)
2. fraction (default): score = phrases found ÷ total must_contain phrases
3. all_required: score = 1.0 only if all phrases found, otherwise 0.0

Examples:

must_contain	Output	Scoring	Score
["ticket #", "reference"]	"Created ticket #1234 as your reference."	fraction	1.0
["ticket #", "reference"]	"Created ticket #1234."	fraction	0.5
["ticket #", "reference"]	"I've noted your issue."	fraction	0.0
["a", "b", "c"]	"a and b present"	all_required	0.0
—	"Sorry we cannot help." (banned phrase)	any	0.0

---

Tier 2 — GEvalMetric with plain English criteria (already built-in)

When you need an LLM to judge something that can't be captured with keyword lists:

- name:       GEvalMetric
  geval_name: "BrandTone"
  criteria:   "The response must use a warm, professional tone. It should not sound dismissive, rushed, or robotic."
  evaluation_params: [input, actual_output]
  threshold:  0.70
  weight:     30

Write what you want in plain English — the LLM evaluates it. No Python file needed.

---

Tier 3 — Custom Python class (developers only)

For checks that need real code — regex, ML models, database lookups, complex scoring:

Step 1 — Create a Python file:

# myapp/metrics/ticket_check.py
from ignis_evaluation.eval_library.metric_registry import BaseMetric, MetricResult
import re

class TicketNumberMetric(BaseMetric):
    """Checks that a support ticket reference (#12345) is included."""

    def __init__(self, threshold: float = 0.7, weight: float = 100, **kwargs):
        self.threshold = float(threshold)
        self.weight    = float(weight)

    def evaluate(self, test_case: dict) -> MetricResult:
        output          = test_case.get("output_text") or ""
        has_ticket_ref  = bool(re.search(r"#\d{4,8}", output))
        has_ticket_word = "ticket" in output.lower()
        score           = 1.0 if (has_ticket_ref and has_ticket_word) else 0.5 if has_ticket_word else 0.0

        return MetricResult(
            score=round(score, 4),
            details={
                "reason":   "Ticket reference found." if score == 1.0 else "Missing ticket number.",
                "has_ref":  has_ticket_ref,
                "has_word": has_ticket_word,
            },
        )

Step 2 — Reference it in the bundle YAML with module::

- name:      TicketNumberMetric
  module:    myapp.metrics.ticket_check.TicketNumberMetric
  threshold: 0.80
  weight:    25

Step 3 — Set IGNIS_BUNDLE_DIR in .env and use the bundle name as usual. Done.

The module: value is the same dotted path you'd use in from myapp.metrics.ticket_check import TicketNumberMetric.

Restart required after adding a new bundle YAML.
Bundle files are loaded once at server startup. If you add a new YAML to IGNIS_BUNDLE_DIR while the server is running, restart uvicorn before calling the API — otherwise the server returns "bundle not found".

evaluate() contract:

Requirement	Detail
Inherit from	ignis_evaluation.eval_library.metric_registry.BaseMetric
Implement	evaluate(self, test_case: dict) -> MetricResult
test_case keys available	input_text, output_text, context, expected_output, retrieval_context
Return	MetricResult(score=float, details=dict) — score must be 0.0–1.0
Constructor	Must accept threshold and weight kwargs from YAML

---

Built-in rule-based metrics (no setup needed)

Metric	Checks	Python file?
KeywordCheckMetric	Any must_contain / must_not_contain lists	No
FinanceComplianceMetric	Banned finance phrases (guaranteed returns, risk-free)	No
MedicalAccuracyMetric	Dangerous medical claims	No
CompositeQualityMetric	Faithfulness + AnswerRelevancy blended	No (needs LLM)
SafetyComplianceMetric	Toxicity + Hallucination blended	No (needs LLM)

---

Project structure (Tier 3 only)

your_project/
├── myapp/
│   ├── __init__.py
│   └── metrics/
│       ├── __init__.py
│       └── ticket_check.py     ← custom metric class
├── my_bundles/
│   └── support_quality.yaml    ← references module: myapp.metrics.ticket_check...
├── .env                        ← IGNIS_BUNDLE_DIR=./my_bundles
└── main.py

---

Troubleshooting

DB write failed: name '_infer_provider' is not defined

The writer module has a function ordering issue. Run pip install --upgrade ignis_evaluation or restart the server after a fresh install.

Metrics return placeholder scores (0.75) instead of real scores

The config or bundle is running in offline mode. Check that OPENAI_API_KEY is set in .env and the config file has use_deepeval: true (for V1 configs).

bundle 'xyz' not found

• Verify the bundle name spelling: GET /bundles returns the full list.
• For custom bundles, confirm IGNIS_BUNDLE_DIR is set and the YAML file exists in that folder.
• Restart the server — bundles load at startup, not per request.

ConnectionError: Cannot connect to Eval API

The eval worker cannot reach the server. Confirm the server is running and EVAL_API_URL is correct in .env.

DB not configured in health check

EVAL_DB_URL is not set or is invalid. Evaluations still work — results go to JSON/CSV files in EVAL_REPORTS_DIR. Set EVAL_DB_URL to enable persistence.

Timeout errors during evaluation

DeepEval metrics can be slow for large batches. Increase timeouts in .env:

DEEPEVAL_PER_TASK_TIMEOUT=600
DEEPEVAL_PER_ATTEMPT_TIMEOUT_SECONDS_OVERRIDE=300

Swagger UI "Try it out" calls fail (CORS error)

Add your Swagger URL to CORS_ALLOWED_ORIGINS:

CORS_ALLOWED_ORIGINS=http://localhost:8001

IGNIS_BUNDLE_STRICT=true prevents server from starting

A custom bundle YAML has validation errors. Check the startup log for the exact error and field. Fix the YAML or set IGNIS_BUNDLE_STRICT=false while developing.

---

Quick Reference — Common Workflows

Evaluate a chatbot response (minimal)

POST /evaluate
{ "bundle": "core_quality", "input": "...", "output": "..." }

Evaluate a RAG response with context

POST /evaluate
{ "bundle": "rag_standard", "input": "...", "output": "...", "context": ["..."] }

Evaluate a full chatbot session

POST /evaluate/conversational
{ "bundle": "chatbot_quality", "turns": [{"user":"...", "assistant":"..."}, ...] }

View dashboard for a project (last 7 days)

POST /eval/dashboard
{ "project": "MyProject", "window_hours": 168 }

Add a custom bundle and use it

# 1. Create ./my_bundles/my_bundle.yaml
# 2. Set IGNIS_BUNDLE_DIR=./my_bundles in .env
# 3. Restart server
# 4. POST /evaluate  { "bundle": "my_bundle", ... }

Run production trace evaluation

python -m ignis_evaluation.workers.eval_worker --project my_project