ignis-evaluation 1.0.1

Config-driven LLM evaluation framework with decorator-based tracing, FastAPI service, and PostgreSQL persistence

ignis_evaluation

A config-driven LLM evaluation framework built on top of DeepEval.
Define metrics in a YAML bundle config, run evaluations via REST API or decorator, persist scores to PostgreSQL, and view them on a hierarchical evaluation dashboard.

---

Table of Contents

• Features
• Project Structure
• Quick Start
• Configuration Reference
• Metric Bundles
• REST API
• Evaluation Dashboard
• Decorator-Driven Evaluation
• Observability Integration
• Database Setup
• Running Tests & Examples
• Environment Variables Reference
• Troubleshooting

---

Features

• YAML bundle configs — group metrics into named bundles; switch bundles per API call with no code change
• Hierarchical dashboard — domain → project → bundles → metrics with KPIs, 6 chart types, and status badges
• Observability integration — eval worker reads production traces from an Observability DB and evaluates them automatically
• Parallel metric execution — all metrics run concurrently via ThreadPoolExecutor
• PostgreSQL persistence — every run, test case, and metric score stored automatically
• @ignis_eval decorator — wrap any LLM function in your app to evaluate output inline
• REST API — trigger evaluations and retrieve results over HTTP
• Confident AI upload — optional push to Confident AI cloud dashboard
• Offline mode — run without an OpenAI key using use_deepeval: false

---

Project Structure

deepeval_demo/
├── src/
│   └── ignis_evaluation/
│       ├── api/
│       │   ├── main.py             # FastAPI entry point, router registration
│       │   ├── models.py           # Pydantic request/response models
│       │   └── routers/
│       │       ├── evaluate.py     # POST /evaluate, /evaluate/batch, /evaluate/dataset
│       │       ├── analytics.py    # POST /eval/dashboard + other analytics endpoints
│       │       ├── projects.py     # GET /projects, GET /projects/{name}/bundles
│       │       ├── results.py      # GET /results, GET /results/{run_id}
│       │       ├── metrics.py      # GET /metrics
│       │       ├── health.py       # GET /health
│       │       └── chat.py         # POST /chat (decorator-driven)
│       ├── configs/                # Built-in YAML configs shipped with the package
│       │   ├── chatapp_config.yaml         # Chat app: 9 bundles, input+output only
│       │   ├── geval_bundle_config.yaml    # GEval custom criteria bundles
│       │   ├── bundle_config.yaml          # General purpose quality+safety bundles
│       │   ├── compliance_bundle_config.yaml  # Finance compliance bundles
│       │   ├── rag_config.yaml             # RAG pipeline metrics
│       │   ├── healthcare_config.yaml      # Healthcare domain metrics
│       │   └── ...
│       ├── db/
│       │   ├── connection.py       # PostgreSQL connection
│       │   ├── writer.py           # INSERT evaluation runs, test & metric results
│       │   ├── reader.py           # SELECT runs for Results API
│       │   └── analytics.py        # Dashboard queries (KPIs, charts, bundles)
│       ├── eval_library/
│       │   ├── runner.py           # Core evaluation engine (EvaluationRunner)
│       │   ├── decorator.py        # @ignis_eval decorator
│       │   ├── scorer.py           # Weighted composite scoring
│       │   ├── metric_registry.py  # Maps metric names → DeepEval classes
│       │   └── ...
│       ├── services/
│       │   └── dashboard_service.py  # build_enriched_dashboard() — usable as a library
│       ├── workers/
│       │   └── eval_worker.py      # Observability → Eval pipeline worker
│       └── sql/
│           └── schema.sql          # Database schema (run once)
├── examples_test/
│   ├── configs/                    # Example YAML configs
│   ├── datasets/                   # Versioned evaluation datasets
│   └── ...
├── docs/
│   ├── EVALUATION_ON_OBSERVABILITY.md
│   ├── OBSERVABILITY_EVAL_APPROACH.md
│   ├── DASHBOARD_UI_WIREFRAME.md
│   └── OBSERVABILITY_EVAL_INTEGRATION.md
├── .env.example
├── pyproject.toml
└── README.md

---

Quick Start

1. Clone & create virtual environment

git clone https://github.com/Infogain-GenAI/ignis_evaluation_deepeval.git
cd deepeval_demo

python -m venv venv

# Windows
venv\Scripts\activate

# macOS / Linux
source venv/bin/activate

2. Install dependencies

pip install -e .          # base install
pip install -e ".[api]"   # adds FastAPI server
pip install -e ".[dev]"   # adds test/dev tools

3. Configure environment

copy .env.example .env    # Windows
cp .env.example .env      # macOS/Linux

Minimum required variables in .env:

OPENAI_API_KEY=sk-...

4. Set up the database

# Create the database first:
# CREATE DATABASE deepeval_results;

# Apply schema:
psql -U postgres -d deepeval_results -f src/ignis_evaluation/sql/schema.sql

5. Start the API server

uvicorn src.ignis_evaluation.api.main:app --reload --port 8001

Interactive docs: http://localhost:8001/docs

6. Run your first evaluation

# Via API (Swagger or curl)
curl -X POST http://localhost:8001/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "config": "chatapp",
    "bundle": "core_quality",
    "input": "What is diabetes?",
    "output": "Diabetes is a chronic condition that affects how the body regulates blood sugar."
  }'

---

Configuration Reference

All evaluation behaviour is controlled by a YAML config file.

project:        chatapp           # Groups runs in DB and dashboard
model:          gpt-4o-mini       # LLM for scoring
use_deepeval:   true              # false = offline/mock mode (no API key needed)

hyperparameters:
  model:        gpt-4o-mini
  temperature:  0.3
  max_tokens:   1024

composite_threshold: 0.65         # Weighted score must exceed this to pass

default_bundle: core_quality      # Used when no bundle is specified in API call

metric_bundles:
  core_quality:
    - name: AnswerRelevancyMetric
      threshold: 0.70
      weight:    40
    - name: GEvalMetric
      geval_name: "Clarity"
      criteria: "The response must be written in clear, plain language."
      evaluation_params: [input, actual_output]
      threshold: 0.65
      weight:    35
    - name: GEvalMetric
      geval_name: "Conciseness"
      criteria: "The response must answer the question without unnecessary content."
      evaluation_params: [input, actual_output]
      threshold: 0.60
      weight:    25

reporting:
  format:     json
  output_dir: reports

---

Metric Bundles

Bundles group metrics under a named key. Pass bundle in any API call to select which group runs.

chatapp_config — Chat application (input + output only, no context needed)

Bundle	Metrics	Use Case
core_quality ★	AnswerRelevancy + Clarity + Conciseness	Default — every message
communication	Tone + Clarity + Conciseness	UX reviews, persona validation
safety_guard	Toxicity + Bias + ProfessionalBoundaries	Public-facing deployments
helpfulness	AnswerRelevancy + Completeness + Actionability	Support bots
customer_support	Helpfulness + Empathy + Specificity	Customer service chat
production_monitor	AnswerRelevancy + Toxicity	Lightest — 100% live traffic
content_policy	Toxicity + Bias + Boundaries + FactualHumility	Pre-release compliance
full_audit	All 8 metrics	Weekly audit, model upgrades
response_quality	AnswerRelevancy + ReasoningClarity + Completeness + FactualHumility	A/B testing

bundle_config — General purpose (requires context for some metrics)

Bundle	Metrics
quality	FaithfulnessMetric + AnswerRelevancyMetric
safety	ToxicityMetric + HallucinationMetric

compliance_bundle_config — Finance domain

Bundle	Metrics
compliance	FinanceComplianceMetric + AnswerRelevancyMetric
compliance_strict	FinanceComplianceMetric + AnswerRelevancyMetric + HallucinationMetric
compliance_balanced	FinanceComplianceMetric + AnswerRelevancyMetric + FaithfulnessMetric

Which metrics need context?

Metric	Needs context?
FaithfulnessMetric	✅ Yes
HallucinationMetric	✅ Yes
ContextualPrecision/Recall/Relevancy	✅ Yes
AnswerRelevancyMetric	❌ No
ToxicityMetric / BiasMetric	❌ No
GEvalMetric (custom criteria)	Depends on evaluation_params in config

---

REST API

Start the server: uvicorn src.ignis_evaluation.api.main:app --reload --port 8001

Evaluation endpoints

Method	Endpoint	Description
POST	/evaluate	Single test case
POST	/evaluate/batch	Multiple test cases
POST	/evaluate/dataset	Entire dataset JSON file

POST /evaluate — with bundle and test_name

{
  "config": "chatapp",
  "bundle": "core_quality",
  "input": "How do I reset my password?",
  "output": "Click Forgot Password on the login screen and follow the reset link.",
  "test_name": "obs_trace_abc123",
  "upload_to_confident": false
}

POST /evaluate/batch — multiple cases

{
  "config": "chatapp",
  "bundle": "customer_support",
  "upload_to_confident": false,
  "cases": [
    {
      "input": "I'm frustrated, my account is locked.",
      "output": "I understand that's frustrating. Let me help — go to Settings > Security > Unlock Account.",
      "test_name": "trace-001"
    },
    {
      "input": "What are your business hours?",
      "output": "We're open Monday to Friday, 9 AM to 6 PM IST.",
      "test_name": "trace-002"
    }
  ]
}

test_name — pass your Observability trace_id here to link eval scores back to production traces.
upload_to_confident — set true to push results to Confident AI (requires CONFIDENT_API_KEY). Default: false.

Dashboard & analytics endpoints

Method	Endpoint	Description
POST	/eval/dashboard	Full dashboard — KPIs, charts, domain → project → bundles
GET	/eval/projects/{name}/bundles	Bundle definitions from YAML config for a project
GET	/eval/projects/{project}/trend	Score trend over time
GET	/eval/metrics/leaderboard	Per-metric rankings
GET	/results	List past runs (paginated)
GET	/results/{run_id}	Full detail for one run
GET	/health	Liveness + readiness

---

Evaluation Dashboard

The dashboard provides a single endpoint that returns everything needed to build a UI: KPIs, 6 chart datasets, and a hierarchical breakdown by domain → project → bundles → metrics.

POST /eval/dashboard

POST /eval/dashboard
{
  "project":              "chatapp",
  "window_hours":         168,
  "include_evaluations":  true,
  "eval_limit":           20
}

All fields are optional. Default: last 2 hours, all projects, no evaluation list.

Response structure:

{
  "generated_at": "2026-07-30T10:00:00Z",
  "window_hours": 168,
  "kpis": {
    "evaluation_count":     87,
    "avg_weighted_score":   0.89,
    "pass_rate_pct":        85.0,
    "total_cost_usd":       0.021,
    "error_count":          4,
    "avg_eval_duration_ms": 12500,
    "models_used":          ["gpt-4o-mini"]
  },
  "charts": {
    "score_over_time":  [{ "date": "2026-07-29", "avg_score": 0.88, "pass_rate_pct": 84 }],
    "pass_fail":        { "passed": 74, "failed": 13 },
    "cost_by_metric":   [{ "metric_name": "AnswerRelevancyMetric", "cost_usd": 0.012 }],
    "score_by_metric":  [{ "metric_name": "Clarity", "avg_score": 0.79 }],
    "score_by_model":   [{ "model": "gpt-4o-mini", "avg_score": 0.89, "run_count": 87 }],
    "cost_trend":       [{ "date": "2026-07-29", "total_cost_usd": 0.021 }]
  },
  "domains": [
    {
      "domain": "Finance",
      "projects": [
        {
          "project_name": "chatapp",
          "avg_weighted_score": 0.89,
          "bundles": [
            {
              "bundle_name": "core_quality",
              "is_default":  true,
              "metrics": [
                { "metric_name": "AnswerRelevancyMetric", "avg_score": 1.0,  "pass_rate_pct": 100, "status": "healthy" },
                { "metric_name": "Clarity",               "avg_score": 0.79, "pass_rate_pct": 60,  "status": "warning" },
                { "metric_name": "Conciseness",           "avg_score": null, "pass_rate_pct": null,"status": "not_evaluated" }
              ]
            }
          ],
          "recent_evaluations": [
            {
              "run_id": 100, "test_name": "obs_trace_abc123",
              "metric_name": "Clarity", "score": 0.49, "passed": false,
              "reason": "Response uses jargon that reduces clarity."
            }
          ]
        }
      ]
    }
  ]
}

Metric status values:

Status	Meaning
healthy	pass_rate_pct ≥ 80%
warning	50% ≤ pass_rate_pct < 80%
needs_attention	pass_rate_pct < 50%
not_evaluated	No runs yet for this metric

GET /eval/projects/{project_name}/bundles

Returns all bundles defined in the YAML config for a project. Used by the frontend to render bundle tabs before overlaying scores from the dashboard.

GET /eval/projects/chatapp/bundles

{
  "project_name": "chatapp",
  "default_bundle": "core_quality",
  "bundles": [
    {
      "bundle_name": "core_quality",
      "is_default": true,
      "metrics": [
        { "metric_name": "AnswerRelevancyMetric", "threshold": 0.70, "weight": 40 },
        { "metric_name": "Clarity",               "threshold": 0.65, "weight": 35 }
      ]
    }
  ]
}

Using dashboard in your own app

from ignis_evaluation.services.dashboard_service import build_enriched_dashboard

result = build_enriched_dashboard(
    project="chatapp",
    window_hours=168,
    include_evaluations=True,
    eval_limit=20,
)
# Returns full dict with kpis, charts, domains — same as POST /eval/dashboard

---

Decorator-Driven Evaluation (@ignis_eval)

Wrap any LLM function in your application to evaluate its output inline — no separate API call needed.

from ignis_evaluation.eval_library.decorator import ignis_eval

@ignis_eval(config="chatapp", bundle="core_quality", include_results=True)
def evaluate_chat(*, input_text: str, output_text: str):
    return {"input_text": input_text, "output_text": output_text}

# In your endpoint:
answer = your_llm_call(user_query)
result = evaluate_chat(input_text=user_query, output_text=answer)
# result["evaluation"]["weighted_score"] → 0.91
# result["evaluation"]["metrics"]["Clarity"]["score"] → 0.84

Save a report file per call — add save_report=True to write a decorator_{project}_{timestamp}.json + .csv to EVAL_REPORTS_DIR after every evaluation:

@ignis_eval(config="chatapp", bundle="core_quality", include_results=True, save_report=True)
def evaluate_chat(*, input_text: str, output_text: str):
    return {"input_text": input_text, "output_text": output_text}

What happens automatically:

1. Loads the bundle config from YAML
2. Runs all metrics in parallel
3. Computes weighted composite score
4. Persists result to PostgreSQL (if EVAL_DB_URL is set)
5. Saves JSON + CSV report file (if save_report=True)
6. Returns answer + evaluation payload

---

Observability Integration

Automatically evaluate every production LLM response stored in an Observability DB without writing a single line of pipeline code.

How it works

Observability DB (read-only)            Evaluation DB
  traces.input + output + trace_id  →   eval_worker.py   →   metric_results
  traces.ingested_at (checkpoint)   ←   eval_worker_checkpoints

The eval_worker reads new traces from the Observability DB since the last checkpoint, evaluates them via POST /evaluate/batch, and stores scores in the Evaluation DB. The trace_id is stored as test_case_name — the only link between both databases.

Setup

Add to .env:

# DeepEval DB — stores eval scores and checkpoints
EVAL_DB_URL=postgresql://postgres:pass@localhost:5432/deepeval_results

# Observability DB — read-only, traces source
OBS_DB_URL=postgresql://postgres:pass@localhost:5432/obs_db

# Eval worker settings
EVAL_API_URL=http://localhost:8001
EVAL_WORKER_BATCH_SIZE=100
EVAL_WORKER_LOOKBACK_DAYS=7
EVAL_WORKER_APP_NAME=ignis

Apply the checkpoint table to the Evaluation DB:

psql -U postgres -d deepeval_results -c "
CREATE TABLE IF NOT EXISTS public.eval_worker_checkpoints (
    id               UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    worker_source    VARCHAR(50) NOT NULL DEFAULT 'eval_worker',
    project_name     VARCHAR(255) NOT NULL,
    start_time       TIMESTAMPTZ NOT NULL,
    end_time         TIMESTAMPTZ,
    status           VARCHAR(20) NOT NULL DEFAULT 'running',
    traces_pulled    INTEGER NOT NULL DEFAULT 0,
    last_eval_at     TIMESTAMPTZ,
    config_used      VARCHAR(255),
    bundle_used      VARCHAR(255),
    error            TEXT,
    application_name VARCHAR(255) NOT NULL DEFAULT 'ignis',
    created_at       TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at       TIMESTAMPTZ NOT NULL DEFAULT NOW()
);"

Running the worker

# All projects — uses last checkpoint per project
python -m ignis_evaluation.workers.eval_worker

# Specific obs project
python -m ignis_evaluation.workers.eval_worker --project langwatch_results

# Override bundle for all projects this run
python -m ignis_evaluation.workers.eval_worker --bundle safety_guard

# Re-process from a specific date (ignores checkpoint)
python -m ignis_evaluation.workers.eval_worker --since 2026-07-01T00:00:00

# Dry run — logs what would happen without calling the API
python -m ignis_evaluation.workers.eval_worker --dry-run

Project → config mapping

The worker maps obs project_name → eval config + bundle. Configure via EVAL_PROJECT_CONFIG_MAP env var:

EVAL_PROJECT_CONFIG_MAP={"langwatch_results": {"config": "chatapp", "bundle": "core_quality"}, "testing_traces": {"config": "chatapp", "bundle": "production_monitor"}}

Unknown projects fall back to chatapp config + core_quality bundle.

Checkpoint behaviour

• Each run saves last_eval_at = traces[-1].ingested_at in eval_worker_checkpoints
• Next run fetches only traces with ingested_at > last_eval_at
• Failed runs do NOT update the checkpoint — they retry from the same point next run
• status = 'success' rows only are used as checkpoints — stale running rows are safely ignored

Linking scores back to traces

-- In Evaluation DB: find scores for a specific production trace
SELECT mr.metric_name, mr.score, mr.passed, mr.reason
FROM metric_results mr
JOIN test_results tr ON tr.test_result_id = mr.test_result_id
WHERE tr.test_case_name = 'your-obs-trace-uuid';

---

Database Setup

Schema (run once)

psql -U postgres -d deepeval_results -f src/ignis_evaluation/sql/schema.sql

Tables

Table	Description
evaluation_runs	One row per eval run — hyperparameters, config, bundle, weighted_score, cost
test_results	One row per test case — input, output, test_case_name (= obs trace_id)
metric_results	One row per metric per test case — score, threshold, passed, reason
projects	One row per project — name, domain, created_at
eval_worker_checkpoints	One row per worker run per obs project — checkpoint, status, traces_pulled

Without a database

The framework works fully without PostgreSQL — results are written to JSON/CSV in examples_test/reports/ instead.

---

Running Tests & Examples

# Full end-to-end demo (DB write + read)
python examples_test/test_db_results_demo.py

# GEval bundle evaluation
python examples_test/test_geval_bundle.py

# RAG evaluation
python examples_test/test_rag_deepeval.py

# Decorator usage example
python examples_test/test_decorator_sample.py

# Run all tests
python -m pytest examples_test/ -v

---

Environment Variables Reference

Variable	Required	Default	Description
OPENAI_API_KEY	Yes (real mode)	—	OpenAI API key for DeepEval scoring
CONFIDENT_API_KEY	No	—	Confident AI key for cloud dashboard upload
DEEPEVAL_TELEMETRY_OPT_OUT	No	—	Set YES to disable anonymous telemetry
EVAL_DB_URL	No	—	DeepEval DB connection string (e.g. postgresql://user:pass@host:5432/deepeval_results)
OBS_DB_URL	No	—	Observability DB DSN (eval worker only)
EVAL_DB_URL	No	—	DeepEval DB connection string (e.g. postgresql://user:pass@host:5432/deepeval_results)
OBS_DB_URL	No	—	Observability DB DSN (eval worker only)
EVAL_API_URL	No	http://localhost:8001	Evaluation API base URL (eval worker)
EVAL_WORKER_BATCH_SIZE	No	100	Max traces per worker batch call
EVAL_WORKER_LOOKBACK_DAYS	No	7	Default lookback when no checkpoint exists
EVAL_PROJECT_CONFIG_MAP	No	—	JSON: obs project → {config, bundle}
EVAL_WORKER_APP_NAME	No	ignis	application_name in checkpoint table
PROJECT_DOMAIN	No	—	Default domain for new projects

---

Troubleshooting

ModuleNotFoundError: ignis_evaluation
Run pip install -e . from the repo root.

OSError or path errors on Windows
Add DEEPEVAL_TELEMETRY_OPT_OUT=YES to .env.

Evaluation takes 5+ minutes
Remove ContextualRelevancyMetric (makes multiple LLM calls per chunk). Use use_deepeval: false for offline runs.

Eval worker shows "0 traces" on every run
Check that --project uses the obs DB project_name, not the YAML config project: field. Run without --project to process all discovered projects automatically.

Dashboard shows not_evaluated for all bundle metrics
The bundle config was loaded from YAML but no evaluations have run for those specific metrics yet. Run /evaluate/batch with the bundle name to populate scores.

CONFIDENT_API_KEY returns "Invalid API key" on upload
Run deepeval login in the terminal to authenticate the session. The upload flag upload_to_confident defaults to false — the framework works fully without it.

Database not connecting

1. Confirm the DB exists: psql -U postgres -l
2. Confirm schema was applied: psql -U postgres -d deepeval_results -c "\dt"
3. Check EVAL_DB_URL is set correctly in .env.

---

License

MIT — see LICENSE for details.