ignis-evaluation 1.0.2

Config-driven LLM evaluation framework with decorator-based tracing, FastAPI service, and PostgreSQL persistence

ignis_evaluation

A config-driven LLM evaluation framework built on top of DeepEval.
Define metrics in a YAML bundle config, run evaluations via REST API or decorator, persist scores to PostgreSQL, and view them on a hierarchical evaluation dashboard.

---

Table of Contents

• Features
• Architecture
• Project Structure
• Quick Start
• Running Tests & Examples
• Configuration Reference
• Metric Bundles
• REST API
• Evaluation Dashboard
• Decorator-Driven Evaluation
• Observability Integration
• Database Setup
• Environment Variables Reference
• Troubleshooting
• References

---

Features

• YAML bundle configs — group metrics into named bundles; switch bundles per API call with no code change
• Hierarchical dashboard — domain → project → bundles → metrics with KPIs, 6 chart types, and status badges
• Observability integration — eval worker reads production traces from an Observability DB and evaluates them automatically
• Parallel metric execution — all metrics run concurrently via ThreadPoolExecutor
• PostgreSQL persistence — every run, test case, and metric score stored automatically
• @ignis_eval decorator — wrap any LLM function in your app to evaluate output inline
• REST API — trigger evaluations and retrieve results over HTTP
• Confident AI upload — optional push to Confident AI cloud dashboard
• Offline mode — run without an OpenAI key using use_deepeval: false

---

Architecture

Evaluation Flow Pattern

                          ignis_evaluation — Evaluation Flow

 ┌──────────────────────────────┐
 │         Your App             │
 │                              │
 │  Option A: REST API call     │
 │  POST /evaluate              │
 │                              │
 │  Option B: @ignis_eval       │
 │  decorator on any function   │
 └──────────────┬───────────────┘
                │
                ▼
 ┌──────────────────────────────┐
 │       EvaluationRunner       │
 │                              │
 │  1. Loads YAML bundle config │
 │  2. Resolves metric classes  │
 │  3. Builds DeepEval objects  │
 └──────────────┬───────────────┘
                │
                ▼
 ┌──────────────────────────────┐       ┌─────────────────────────────┐
 │   ThreadPoolExecutor         │──────▶│   OpenAI / LLM Judge        │
 │                              │       │   (gpt-4o-mini or gpt-4o)   │
 │  All metrics run in parallel │◀──────│   Scores each metric        │
 │  AnswerRelevancy · GEval     │ scores│   Returns score + reason    │
 │  Faithfulness · Hallucination│       └─────────────────────────────┘
 └──────────────┬───────────────┘
                │  weighted score + per-metric results
                ▼
 ┌──────────────────────────────┐
 │    EvaluationRunner          │
 │    (collects & aggregates)   │
 └──────┬───────────────┬───────┘
        │               │
        ▼               ▼
 ┌─────────────┐  ┌─────────────────┐
 │  PostgreSQL │  │  JSON + CSV     │
 │  eval runs  │  │  reports/       │
 │  test cases │  │  (local files)  │
 │  metrics    │  └─────────────────┘
 └─────────────┘

Observability Integration Pattern

                          Observability → Evaluation Pipeline

 ┌─────────────────────┐     writes     ┌──────────────────────┐
 │   Production App    │───────────────▶│   Observability DB   │
 │   (Your LLM)        │    traces      │   traces table       │
 └─────────────────────┘                └──────────┬───────────┘
                                                    │
                                          polls for new traces
                                          (ingested_at > last checkpoint)
                                                    │
                                                    ▼
                                         ┌──────────────────────┐
                                         │     Eval Worker      │
                                         │   eval_worker.py     │
                                         │                      │
                                         │  Reads new traces    │
                                         │  Saves checkpoint    │
                                         │  Batches to API      │
                                         └──────────┬───────────┘
                                                    │
                                          POST /evaluate/batch
                                                    │
                                                    ▼
                                         ┌──────────────────────┐
                                         │   ignis_evaluation   │
                                         │   REST API           │
                                         │   (EvaluationRunner) │
                                         └──────────┬───────────┘
                                                    │
                                  ┌─────────────────┴──────────────────┐
                                  ▼                                     ▼
                       ┌──────────────────┐               ┌────────────────────────┐
                       │   PostgreSQL     │               │   Confident AI         │
                       │   Evaluation DB  │               │   (Cloud Dashboard)    │
                       │   metric_results │               │   optional upload      │
                       │   trace_id link  │               └────────────────────────┘
                       └──────────────────┘

Tech Stack

Layer	Technology	Purpose
Language	Python 3.10+	Modern typing, match statements, asyncio support
API Framework	FastAPI + Uvicorn	High-performance async REST API with auto Swagger docs
Evaluation Engine	DeepEval	Built-in RAG, GEval, hallucination, and toxicity metrics
LLM Judge	OpenAI (gpt-4o-mini)	Scores responses via structured LLM calls
Concurrency	ThreadPoolExecutor	Runs all metrics in parallel per evaluation request
Database	PostgreSQL	Persists runs, test cases, and per-metric scores
ORM / Driver	psycopg2	Direct PostgreSQL connection and query execution
Config	YAML (PyYAML)	Declarative metric bundles, thresholds, and weights
Validation	Pydantic v2	Type-safe request/response models for the API
Observability	Confident AI (optional)	Cloud dashboard upload via CONFIDENT_API_KEY
Reporting	JSON + CSV	Local report files written to reports/ per run
Testing	pytest + httpx	Unit and integration tests via FastAPI TestClient

---

Project Structure

deepeval_demo/
├── src/
│   └── ignis_evaluation/
│       ├── api/
│       │   ├── main.py             # FastAPI entry point, router registration
│       │   ├── models.py           # Pydantic request/response models
│       │   └── routers/
│       │       ├── evaluate.py     # POST /evaluate, /evaluate/batch, /evaluate/dataset
│       │       ├── analytics.py    # POST /eval/dashboard + other analytics endpoints
│       │       ├── projects.py     # GET /projects, GET /projects/{name}/bundles
│       │       ├── results.py      # GET /results, GET /results/{run_id}
│       │       ├── metrics.py      # GET /metrics
│       │       ├── health.py       # GET /health
│       │       └── chat.py         # POST /chat (decorator-driven)
│       ├── configs/                # Built-in YAML configs shipped with the package
│       │   ├── chatapp_config.yaml         # Chat app: 9 bundles, input+output only
│       │   ├── geval_bundle_config.yaml    # GEval custom criteria bundles
│       │   ├── bundle_config.yaml          # General purpose quality+safety bundles
│       │   ├── compliance_bundle_config.yaml  # Finance compliance bundles
│       │   ├── rag_config.yaml             # RAG pipeline metrics
│       │   ├── healthcare_config.yaml      # Healthcare domain metrics
│       │   └── ...
│       ├── db/
│       │   ├── connection.py       # PostgreSQL connection
│       │   ├── writer.py           # INSERT evaluation runs, test & metric results
│       │   ├── reader.py           # SELECT runs for Results API
│       │   └── analytics.py        # Dashboard queries (KPIs, charts, bundles)
│       ├── eval_library/
│       │   ├── runner.py           # Core evaluation engine (EvaluationRunner)
│       │   ├── decorator.py        # @ignis_eval decorator
│       │   ├── scorer.py           # Weighted composite scoring
│       │   ├── metric_registry.py  # Maps metric names → DeepEval classes
│       │   └── ...
│       ├── services/
│       │   └── dashboard_service.py  # build_enriched_dashboard() — usable as a library
│       ├── workers/
│       │   └── eval_worker.py      # Observability → Eval pipeline worker
│       └── sql/
│           └── schema.sql          # Database schema (run once)
├── examples_test/
│   ├── configs/                    # Example YAML configs
│   ├── datasets/                   # Versioned evaluation datasets
│   └── ...
├── docs/
│   ├── EVALUATION_ON_OBSERVABILITY.md
│   ├── OBSERVABILITY_EVAL_APPROACH.md
│   ├── DASHBOARD_UI_WIREFRAME.md
│   └── OBSERVABILITY_EVAL_INTEGRATION.md
├── .env.example
├── pyproject.toml
└── README.md

---

Quick Start

1. Clone & create virtual environment

git clone https://github.com/Infogain-GenAI/ignis_evaluation_deepeval.git
cd deepeval_demo

python -m venv venv

# Windows
venv\Scripts\activate

# macOS / Linux
source venv/bin/activate

2. Install dependencies

pip install -e .          # base install
pip install -e ".[api]"   # adds FastAPI server
pip install -e ".[dev]"   # adds test/dev tools

3. Configure environment

copy .env.example .env    # Windows
cp .env.example .env      # macOS/Linux

Minimum required variables in .env:

OPENAI_API_KEY=sk-...

4. Set up the database

# Create the database first:
# CREATE DATABASE deepeval_results;

# Apply schema:
psql -U postgres -d deepeval_results -f src/ignis_evaluation/sql/schema.sql

5. Start the API server

uvicorn src.ignis_evaluation.api.main:app --reload --port 8001

Interactive docs: http://localhost:8001/docs

6. Run your first evaluation

curl -X POST http://localhost:8001/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "config": "chatapp",
    "bundle": "core_quality",
    "input": "What is diabetes?",
    "output": "Diabetes is a chronic condition that affects how the body regulates blood sugar."
  }'

You should see a JSON response with weighted_score, passed, and per-metric results.

---

Running Tests & Examples

Three copy-paste examples to verify the library is working. Start with Example 1 (no API key needed), then progress as needed.

Prerequisites

pip install -e ".[dev]"

---

Example 1 — Offline evaluation (no API key needed)

Uses TestClient to run the FastAPI app in-process — no server to start, no network calls, instant. Setting use_deepeval: false skips all LLM calls so no OpenAI key or database is required.

Step 1 — Create the config file (my_offline_config.yaml):

# my_offline_config.yaml
project:             my_project
model:               gpt-4o-mini    # LLM judge (not called when use_deepeval: false)
use_deepeval:        false          # offline mode — returns placeholder scores instantly

composite_threshold: 0.65

metrics:
  - name:      FaithfulnessMetric   # checks answer is grounded in context (no hallucination)
    threshold: 0.80
    weight:    40

  - name:      AnswerRelevancyMetric  # checks answer directly addresses the question
    threshold: 0.75
    weight:    35

  - name:      HallucinationMetric   # penalises claims not in the retrieved context
    threshold: 0.80
    weight:    25

reporting:
  format:     json
  output_dir: reports

The built-in equivalent is src/ignis_evaluation/configs/rag_config.yaml. Change use_deepeval: true there to go live with real LLM scoring.

Step 2 — Run the evaluation:

python examples_test/test_api.py

Or paste this into a Python file and run it:

# If you installed via `pip install ignis_evaluation`, remove the two lines below.
# They are only needed when running from the cloned repo with `pip install -e .`
import sys
from pathlib import Path
sys.path.insert(0, str(Path(".") / "src"))

from fastapi.testclient import TestClient
from ignis_evaluation.api.main import app

client = TestClient(app)

# Health check
print(client.get("/health").json())

# Offline evaluation — use_deepeval: false means no OpenAI call, runs instantly
result = client.post("/evaluate", json={
    "config": "rag",                 # loads src/ignis_evaluation/configs/rag_config.yaml
                                     # or pass config_path to use your own file
    "input": "What is Python?",
    "output": "Python is a high-level programming language.",
    "context": ["Python is a high-level, general-purpose programming language."],
}).json()
print("Score :", result.get("weighted_score"))
print("Passed:", result.get("passed"))
for name, m in result.get("metrics", {}).items():
    status = "PASS" if m.get("pass") else "FAIL"
    print(f"  {name}: {m.get('score')} — {status}")

Expected output (runs in seconds):

{'status': 'ok', 'db': 'not_configured', ...}
Score : 0.75
Passed: True
  FaithfulnessMetric: 0.75 — PASS
  AnswerRelevancyMetric: 0.75 — PASS
  HallucinationMetric: 0.75 — PASS

---

Example 2 — GEval custom criteria (requires OPENAI_API_KEY)

GEval lets you define your own rubric criteria instead of using fixed metrics. Each criterion is a plain-English sentence describing what "good" means for your use case.

Step 1 — Create the config file (my_geval_config.yaml):

# my_geval_config.yaml
project:             my_project
model:               gpt-4o-mini   # LLM used as the judge
use_deepeval:        true          # false = offline/mock mode (no API key needed)
composite_threshold: 0.65          # weighted score must exceed this to pass

metrics:
  - name:       GEvalMetric
    geval_name: "Correctness"      # label shown in results table
    criteria: >
      The actual output must be factually correct and consistent with
      the provided context. Any claim not supported by the context
      should be penalised.
    evaluation_params: [input, actual_output, context]
    evaluation_steps:
      - "Check if every factual claim is supported by the context."
      - "Penalise any claim that contradicts or is absent from the context."
      - "Score 0–1: 1 = fully correct, 0 = entirely wrong."
    threshold: 0.70
    weight:    40

  - name:       GEvalMetric
    geval_name: "Clarity"
    criteria: >
      The actual output must be written in clear, plain language that
      a non-expert can understand. Avoid unnecessary jargon.
    evaluation_params: [input, actual_output]
    evaluation_steps:
      - "Is the language simple and accessible to a general audience?"
      - "Is the response free from unnecessary technical jargon?"
      - "Is it logically structured with a clear point?"
    threshold: 0.65
    weight:    35

  - name:       GEvalMetric
    geval_name: "Conciseness"
    criteria: >
      The actual output must answer the question without unnecessary
      repetition, padding, or off-topic content.
    evaluation_params: [input, actual_output]
    evaluation_steps:
      - "Does the response stay on topic?"
      - "Is there repetition of the same point?"
      - "Is the length appropriate for the question asked?"
    threshold: 0.60
    weight:    25

The full working example is at examples_test/configs/geval_config.yaml.

Step 2 — Run the evaluation:

python examples_test/test_geval.py

Or paste this into a Python file:

# If you installed via `pip install ignis_evaluation`, the sys.path lines are not needed.
# Only uncomment them if running from the cloned repo without pip install -e .
# import sys; from pathlib import Path
# sys.path.insert(0, str(Path(".") / "src"))

from pathlib import Path
from dotenv import load_dotenv
load_dotenv(".env")  # loads OPENAI_API_KEY

# Pre-import openai before ThreadPoolExecutor starts — prevents Python 3.12
# import-lock deadlock when parallel metric threads first-import openai.
import openai  # noqa: F401
import openai.resources.chat  # noqa: F401

from ignis_evaluation.eval_library.runner import EvaluationRunner

# Point to your config file (or use the provided example)
config_path = Path("my_geval_config.yaml")  # or: Path("examples_test/configs/geval_config.yaml")
runner = EvaluationRunner(config_path)

result = runner.run(
    input_text="What is photosynthesis?",
    actual_output=(
        "Photosynthesis is the process by which plants convert sunlight, "
        "water, and carbon dioxide into glucose and oxygen."
    ),
    context=[
        "Photosynthesis converts light energy into chemical energy stored as glucose.",
        "It requires sunlight, water (H2O), and carbon dioxide (CO2).",
        "Oxygen (O2) is released as a byproduct.",
    ],
)
print("Score:", result.get("weighted_score"))
for name, m in result.get("metrics", {}).items():
    status = "PASS" if m.get("pass") else "FAIL"
    print(f"  {name}: {m.get('score')} — {status}")

Expected output:

Score: 0.91
  Correctness: 0.95 — PASS
  Clarity: 0.88 — PASS
  Conciseness: 0.89 — PASS

---

Example 3 — Decorator (@ignis_eval)

The @ignis_eval decorator wraps any LLM function to evaluate its output inline using a bundle — a named group of metrics defined in a config file. No separate API call needed.

Step 1 — Understand the config (src/ignis_evaluation/configs/chatapp_config.yaml):

# chatapp_config.yaml  (excerpt — the built-in config used by config="chatapp")
project:             chatapp
model:               gpt-4o
use_deepeval:        true
composite_threshold: 0.65
default_bundle:      core_quality

metric_bundles:

  core_quality:           # <-- bundle name passed to @ignis_eval
    - name:       AnswerRelevancyMetric
      threshold:  0.70
      weight:     40

    - name:       GEvalMetric
      geval_name: "Clarity"
      criteria: >
        The response must be written in clear, plain language that a
        non-expert can understand. Avoid jargon and be logically structured.
      evaluation_params: [input, actual_output]
      threshold:  0.65
      weight:     35

    - name:       GEvalMetric
      geval_name: "Conciseness"
      criteria: >
        The response must answer the question without unnecessary
        repetition, padding, or off-topic content.
      evaluation_params: [input, actual_output]
      threshold:  0.60
      weight:     25

To use your own config, pass config_path=Path("my_config.yaml") instead of config="chatapp".
To add more bundles, add new keys under metric_bundles: in your YAML.

Step 2 — Run the decorator:

python examples_test/test_decorator_sample.py

Or paste this into a Python file:

# If you installed via `pip install ignis_evaluation`, the sys.path lines are not needed.
# Only uncomment them if running from the cloned repo without pip install -e .
# import sys; from pathlib import Path
# sys.path.insert(0, str(Path(".") / "src"))

from dotenv import load_dotenv
load_dotenv(".env")  # loads OPENAI_API_KEY

# Pre-import openai before ThreadPoolExecutor starts — prevents Python 3.12
# import-lock deadlock when parallel metric threads first-import openai.
import openai  # noqa: F401
import openai.resources.chat  # noqa: F401

from ignis_evaluation.eval_library.decorator import ignis_eval

# config="chatapp"  → loads src/ignis_evaluation/configs/chatapp_config.yaml
# bundle="core_quality" → runs the 3 metrics defined in that bundle
@ignis_eval(config="chatapp", bundle="core_quality", include_results=True)
def evaluate_response(*, input_text: str, output_text: str):
    return {"input_text": input_text, "output_text": output_text}

result = evaluate_response(
    input_text="How do I reset my password?",
    output_text="Click 'Forgot Password' on the login screen and follow the reset link sent to your email.",
)

eval_data = result["evaluation"]
all_passed = all(m.get("pass", False) for m in eval_data.get("metrics", {}).values() if isinstance(m, dict))
print("Score :", eval_data.get("weighted_score"))
print("Passed:", all_passed)
for name, m in eval_data.get("metrics", {}).items():
    status = "PASS" if m.get("pass") else "FAIL"
    print(f"  {name}: {m.get('score')} — {status}")

Expected output:

Score : 0.97
Passed: True
  Clarity: 0.96 — PASS
  Conciseness: 0.93 — PASS
  AnswerRelevancyMetric: 1.0 — PASS

---

Run all tests

python -m pytest examples_test/ -v

---

Configuration Reference

All evaluation behaviour is controlled by a YAML config file.

project:        chatapp           # Groups runs in DB and dashboard
model:          gpt-4o-mini       # LLM for scoring
use_deepeval:   true              # false = offline/mock mode (no API key needed)

hyperparameters:
  model:        gpt-4o-mini
  temperature:  0.3
  max_tokens:   1024

composite_threshold: 0.65         # Weighted score must exceed this to pass

default_bundle: core_quality      # Used when no bundle is specified in API call

metric_bundles:
  core_quality:
    - name: AnswerRelevancyMetric
      threshold: 0.70
      weight:    40
    - name: GEvalMetric
      geval_name: "Clarity"
      criteria: "The response must be written in clear, plain language."
      evaluation_params: [input, actual_output]
      threshold: 0.65
      weight:    35
    - name: GEvalMetric
      geval_name: "Conciseness"
      criteria: "The response must answer the question without unnecessary content."
      evaluation_params: [input, actual_output]
      threshold: 0.60
      weight:    25

reporting:
  format:     json
  output_dir: reports

---

Metric Bundles

Bundles group metrics under a named key. Pass bundle in any API call to select which group runs.

chatapp_config — Chat application (input + output only, no context needed)

Bundle	Metrics	Use Case
core_quality ★	AnswerRelevancy + Clarity + Conciseness	Default — every message
communication	Tone + Clarity + Conciseness	UX reviews, persona validation
safety_guard	Toxicity + Bias + ProfessionalBoundaries	Public-facing deployments
helpfulness	AnswerRelevancy + Completeness + Actionability	Support bots
customer_support	Helpfulness + Empathy + Specificity	Customer service chat
production_monitor	AnswerRelevancy + Toxicity	Lightest — 100% live traffic
content_policy	Toxicity + Bias + Boundaries + FactualHumility	Pre-release compliance
full_audit	All 8 metrics	Weekly audit, model upgrades
response_quality	AnswerRelevancy + ReasoningClarity + Completeness + FactualHumility	A/B testing

bundle_config — General purpose (requires context for some metrics)

Bundle	Metrics
quality	FaithfulnessMetric + AnswerRelevancyMetric
safety	ToxicityMetric + HallucinationMetric

compliance_bundle_config — Finance domain

Bundle	Metrics
compliance	FinanceComplianceMetric + AnswerRelevancyMetric
compliance_strict	FinanceComplianceMetric + AnswerRelevancyMetric + HallucinationMetric
compliance_balanced	FinanceComplianceMetric + AnswerRelevancyMetric + FaithfulnessMetric

Which metrics need context?

Metric	Needs context?
FaithfulnessMetric	✅ Yes
HallucinationMetric	✅ Yes
ContextualPrecision/Recall/Relevancy	✅ Yes
AnswerRelevancyMetric	❌ No
ToxicityMetric / BiasMetric	❌ No
GEvalMetric (custom criteria)	Depends on evaluation_params in config

---

REST API

Start the server: uvicorn src.ignis_evaluation.api.main:app --reload --port 8001

Evaluation endpoints

Method	Endpoint	Description
POST	/evaluate	Single test case
POST	/evaluate/batch	Multiple test cases
POST	/evaluate/dataset	Entire dataset JSON file

POST /evaluate — with bundle and test_name

{
  "config": "chatapp",
  "bundle": "core_quality",
  "input": "How do I reset my password?",
  "output": "Click Forgot Password on the login screen and follow the reset link.",
  "test_name": "obs_trace_abc123",
  "upload_to_confident": false
}

POST /evaluate/batch — multiple cases

{
  "config": "chatapp",
  "bundle": "customer_support",
  "upload_to_confident": false,
  "cases": [
    {
      "input": "I'm frustrated, my account is locked.",
      "output": "I understand that's frustrating. Let me help — go to Settings > Security > Unlock Account.",
      "test_name": "trace-001"
    },
    {
      "input": "What are your business hours?",
      "output": "We're open Monday to Friday, 9 AM to 6 PM IST.",
      "test_name": "trace-002"
    }
  ]
}

test_name — pass your Observability trace_id here to link eval scores back to production traces.
upload_to_confident — set true to push results to Confident AI (requires CONFIDENT_API_KEY). Default: false.

Dashboard & analytics endpoints

Method	Endpoint	Description
POST	/eval/dashboard	Full dashboard — KPIs, charts, domain → project → bundles
GET	/eval/projects/{name}/bundles	Bundle definitions from YAML config for a project
GET	/eval/projects/{project}/trend	Score trend over time
GET	/eval/metrics/leaderboard	Per-metric rankings
GET	/results	List past runs (paginated)
GET	/results/{run_id}	Full detail for one run
GET	/health	Liveness + readiness

---

Evaluation Dashboard

The dashboard provides a single endpoint that returns everything needed to build a UI: KPIs, 6 chart datasets, and a hierarchical breakdown by domain → project → bundles → metrics.

POST /eval/dashboard

POST /eval/dashboard
{
  "project":              "chatapp",
  "window_hours":         168,
  "include_evaluations":  true,
  "eval_limit":           20
}

All fields are optional. Default: last 2 hours, all projects, no evaluation list.

Response structure:

{
  "generated_at": "2026-07-30T10:00:00Z",
  "window_hours": 168,
  "kpis": {
    "evaluation_count":     87,
    "avg_weighted_score":   0.89,
    "pass_rate_pct":        85.0,
    "total_cost_usd":       0.021,
    "error_count":          4,
    "avg_eval_duration_ms": 12500,
    "models_used":          ["gpt-4o-mini"]
  },
  "charts": {
    "score_over_time":  [{ "date": "2026-07-29", "avg_score": 0.88, "pass_rate_pct": 84 }],
    "pass_fail":        { "passed": 74, "failed": 13 },
    "cost_by_metric":   [{ "metric_name": "AnswerRelevancyMetric", "cost_usd": 0.012 }],
    "score_by_metric":  [{ "metric_name": "Clarity", "avg_score": 0.79 }],
    "score_by_model":   [{ "model": "gpt-4o-mini", "avg_score": 0.89, "run_count": 87 }],
    "cost_trend":       [{ "date": "2026-07-29", "total_cost_usd": 0.021 }]
  },
  "domains": [
    {
      "domain": "Finance",
      "projects": [
        {
          "project_name": "chatapp",
          "avg_weighted_score": 0.89,
          "bundles": [
            {
              "bundle_name": "core_quality",
              "is_default":  true,
              "metrics": [
                { "metric_name": "AnswerRelevancyMetric", "avg_score": 1.0,  "pass_rate_pct": 100, "status": "healthy" },
                { "metric_name": "Clarity",               "avg_score": 0.79, "pass_rate_pct": 60,  "status": "warning" },
                { "metric_name": "Conciseness",           "avg_score": null, "pass_rate_pct": null,"status": "not_evaluated" }
              ]
            }
          ],
          "recent_evaluations": [
            {
              "run_id": 100, "test_name": "obs_trace_abc123",
              "metric_name": "Clarity", "score": 0.49, "passed": false,
              "reason": "Response uses jargon that reduces clarity."
            }
          ]
        }
      ]
    }
  ]
}

Metric status values:

Status	Meaning
healthy	pass_rate_pct ≥ 80%
warning	50% ≤ pass_rate_pct < 80%
needs_attention	pass_rate_pct < 50%
not_evaluated	No runs yet for this metric

GET /eval/projects/{project_name}/bundles

Returns all bundles defined in the YAML config for a project. Used by the frontend to render bundle tabs before overlaying scores from the dashboard.

GET /eval/projects/chatapp/bundles

{
  "project_name": "chatapp",
  "default_bundle": "core_quality",
  "bundles": [
    {
      "bundle_name": "core_quality",
      "is_default": true,
      "metrics": [
        { "metric_name": "AnswerRelevancyMetric", "threshold": 0.70, "weight": 40 },
        { "metric_name": "Clarity",               "threshold": 0.65, "weight": 35 }
      ]
    }
  ]
}

Using dashboard in your own app

from ignis_evaluation.services.dashboard_service import build_enriched_dashboard

result = build_enriched_dashboard(
    project="chatapp",
    window_hours=168,
    include_evaluations=True,
    eval_limit=20,
)
# Returns full dict with kpis, charts, domains — same as POST /eval/dashboard

---

Decorator-Driven Evaluation (@ignis_eval)

Wrap any LLM function in your application to evaluate its output inline — no separate API call needed.

from ignis_evaluation.eval_library.decorator import ignis_eval

@ignis_eval(config="chatapp", bundle="core_quality", include_results=True)
def evaluate_chat(*, input_text: str, output_text: str):
    return {"input_text": input_text, "output_text": output_text}

# In your endpoint:
answer = your_llm_call(user_query)
result = evaluate_chat(input_text=user_query, output_text=answer)
# result["evaluation"]["weighted_score"] → 0.91
# result["evaluation"]["metrics"]["Clarity"]["score"] → 0.84

Save a report file per call — add save_report=True to write a decorator_{project}_{timestamp}.json + .csv to EVAL_REPORTS_DIR after every evaluation:

@ignis_eval(config="chatapp", bundle="core_quality", include_results=True, save_report=True)
def evaluate_chat(*, input_text: str, output_text: str):
    return {"input_text": input_text, "output_text": output_text}

What happens automatically:

1. Loads the bundle config from YAML
2. Runs all metrics in parallel
3. Computes weighted composite score
4. Persists result to PostgreSQL (if EVAL_DB_URL is set)
5. Saves JSON + CSV report file (if save_report=True)
6. Returns answer + evaluation payload

---

Observability Integration

Automatically evaluate every production LLM response stored in an Observability DB without writing a single line of pipeline code.

How it works

Observability DB (read-only)            Evaluation DB
  traces.input + output + trace_id  →   eval_worker.py   →   metric_results
  traces.ingested_at (checkpoint)   ←   eval_worker_checkpoints

The eval_worker reads new traces from the Observability DB since the last checkpoint, evaluates them via POST /evaluate/batch, and stores scores in the Evaluation DB. The trace_id is stored as test_case_name — the only link between both databases.

Setup

Add to .env:

# DeepEval DB — stores eval scores and checkpoints
EVAL_DB_URL=postgresql://postgres:pass@localhost:5432/deepeval_results

# Observability DB — read-only, traces source
OBS_DB_URL=postgresql://postgres:pass@localhost:5432/obs_db

# Eval worker settings
EVAL_API_URL=http://localhost:8001
EVAL_WORKER_BATCH_SIZE=100
EVAL_WORKER_LOOKBACK_DAYS=7
EVAL_WORKER_APP_NAME=ignis

Apply the checkpoint table to the Evaluation DB:

psql -U postgres -d deepeval_results -c "
CREATE TABLE IF NOT EXISTS public.eval_worker_checkpoints (
    id               UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    worker_source    VARCHAR(50) NOT NULL DEFAULT 'eval_worker',
    project_name     VARCHAR(255) NOT NULL,
    start_time       TIMESTAMPTZ NOT NULL,
    end_time         TIMESTAMPTZ,
    status           VARCHAR(20) NOT NULL DEFAULT 'running',
    traces_pulled    INTEGER NOT NULL DEFAULT 0,
    last_eval_at     TIMESTAMPTZ,
    config_used      VARCHAR(255),
    bundle_used      VARCHAR(255),
    error            TEXT,
    application_name VARCHAR(255) NOT NULL DEFAULT 'ignis',
    created_at       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);"

Running the worker

# All projects — uses last checkpoint per project
python -m ignis_evaluation.workers.eval_worker

# Specific obs project
python -m ignis_evaluation.workers.eval_worker --project langwatch_results

# Override bundle for all projects this run
python -m ignis_evaluation.workers.eval_worker --bundle safety_guard

# Re-process from a specific date (ignores checkpoint)
python -m ignis_evaluation.workers.eval_worker --since 2026-07-01T00:00:00

# Dry run — logs what would happen without calling the API
python -m ignis_evaluation.workers.eval_worker --dry-run

Project → config mapping

The worker maps obs project_name → eval config + bundle. Configure via EVAL_PROJECT_CONFIG_MAP env var:

EVAL_PROJECT_CONFIG_MAP={"langwatch_results": {"config": "chatapp", "bundle": "core_quality"}, "testing_traces": {"config": "chatapp", "bundle": "production_monitor"}}

Unknown projects fall back to chatapp config + core_quality bundle.

Checkpoint behaviour

• Each run saves last_eval_at = traces[-1].ingested_at in eval_worker_checkpoints
• Next run fetches only traces with ingested_at > last_eval_at
• Failed runs do NOT update the checkpoint — they retry from the same point next run
• status = 'success' rows only are used as checkpoints — stale running rows are safely ignored

Linking scores back to traces

-- In Evaluation DB: find scores for a specific production trace
SELECT mr.metric_name, mr.score, mr.passed, mr.reason
FROM metric_results mr
JOIN test_results tr ON tr.test_result_id = mr.test_result_id
WHERE tr.test_case_name = 'your-obs-trace-uuid';

---

Database Setup

Schema (run once)

psql -U postgres -d deepeval_results -f src/ignis_evaluation/sql/schema.sql

Tables

Table	Description
evaluation_runs	One row per eval run — hyperparameters, config, bundle, weighted_score, cost
test_results	One row per test case — input, output, test_case_name (= obs trace_id)
metric_results	One row per metric per test case — score, threshold, passed, reason
projects	One row per project — name, domain, created_at
eval_worker_checkpoints	One row per worker run per obs project — checkpoint, status, traces_pulled

Without a database

The framework works fully without PostgreSQL — results are written to JSON/CSV in examples_test/reports/ instead.

---

Environment Variables Reference

Variable	Required	Default	Description
OPENAI_API_KEY	Yes (real mode)	—	OpenAI API key for DeepEval scoring
CONFIDENT_API_KEY	No	—	Confident AI key for cloud dashboard upload
DEEPEVAL_TELEMETRY_OPT_OUT	No	—	Set YES to disable anonymous telemetry
EVAL_DB_URL	No	—	DeepEval DB connection string (e.g. postgresql://user:pass@host:5432/deepeval_results)
OBS_DB_URL	No	—	Observability DB DSN (eval worker only)
EVAL_API_URL	No	http://localhost:8001	Evaluation API base URL (eval worker)
EVAL_WORKER_BATCH_SIZE	No	100	Max traces per worker batch call
EVAL_WORKER_LOOKBACK_DAYS	No	7	Default lookback when no checkpoint exists
EVAL_PROJECT_CONFIG_MAP	No	—	JSON: obs project → {config, bundle}
EVAL_WORKER_APP_NAME	No	ignis	application_name in checkpoint table
PROJECT_DOMAIN	No	—	Default domain for new projects

---

Troubleshooting

ModuleNotFoundError: ignis_evaluation
Run pip install -e . from the repo root.

OSError or path errors on Windows
Add DEEPEVAL_TELEMETRY_OPT_OUT=YES to .env.

Evaluation takes 5+ minutes
Remove ContextualRelevancyMetric (makes multiple LLM calls per chunk). Use use_deepeval: false for offline runs.

Eval worker shows "0 traces" on every run
Check that --project uses the obs DB project_name, not the YAML config project: field. Run without --project to process all discovered projects automatically.

Dashboard shows not_evaluated for all bundle metrics
The bundle config was loaded from YAML but no evaluations have run for those specific metrics yet. Run /evaluate/batch with the bundle name to populate scores.

CONFIDENT_API_KEY returns "Invalid API key" on upload
Run deepeval login in the terminal to authenticate the session. The upload flag upload_to_confident defaults to false — the framework works fully without it.

Database not connecting

1. Confirm the DB exists: psql -U postgres -l
2. Confirm schema was applied: psql -U postgres -d deepeval_results -c "\dt"
3. Check EVAL_DB_URL is set correctly in .env.

---

References

Built with:

• DeepEval — LLM evaluation framework powering all metrics
• Confident AI — Cloud dashboard for evaluation results and dataset management
• OpenAI — LLM judge provider (gpt-4o-mini, gpt-4o)
• FastAPI — REST API framework with auto Swagger docs
• Uvicorn — ASGI server for the evaluation API
• PostgreSQL — Evaluation results persistence
• psycopg2 — PostgreSQL driver
• Pydantic — Request/response validation and settings
• PyYAML — YAML config file parsing
• python-dotenv — .env file loading
• Rich — Terminal metric result tables

---

Questions? Open an issue or check the examples_test/ directory for copy-paste examples.

Ready to evaluate your LLM? Start with examples_test/test_api.py — no API key needed.