evalytic 0.4.0

Evaluation SDK for AI outputs

Evalytic

Evals for visual AI. Automated quality evaluation for AI-generated images and video.

Know if your AI-generated visuals are good before your users tell you they're not.

pip install evalytic

# Score any image you already have
evaly eval --image output.png --prompt "A sunset over mountains" --yes

# Compare models side by side
evaly bench -m flux-schnell -m flux-dev -m flux-pro \
  -p "A product photo on marble countertop" --yes

What It Does

Evalytic scores AI-generated images using two complementary approaches:

• VLM Judges (Gemini, GPT, Claude, Ollama) evaluate semantic dimensions like prompt adherence, text rendering, and identity preservation
• Local Metrics (sharpness, CLIP, LPIPS, ArcFace) run on your machine, free, no API key needed

Use both together or either one alone. Works with any image, from any provider or your own pipeline.

Use Cases

• Model Selection - Compare models with real prompts, pick the best one for your use case
• Prompt Optimization - Measure how well models follow your prompts across dimensions
• Regression Detection - Catch quality drops when models or prompts update
• CI/CD Quality Gate - Block deploys when image quality falls below threshold
• 7 Semantic Dimensions - visual_quality, prompt_adherence, text_rendering, input_fidelity, transformation_quality, artifact_detection, identity_preservation
• Consensus Judging - Multi-judge scoring with automatic agreement analysis

Quickstart

1. Install

pip install evalytic

2. See Real Examples (no API key needed)

evaly demo              # Opens showcase with 4 real benchmark case studies
evaly demo face         # Face identity preservation comparison
evaly demo flagship     # Flux Schnell vs Dev vs Pro cost/quality

3. Score an Existing Image

# Local metrics only (free, no API key)
evaly eval --image output.png --prompt "A sunset over mountains" --no-judge

# With VLM judge
export GEMINI_API_KEY=your_gemini_key
evaly eval --image output.png --prompt "A sunset over mountains" --yes

4. Benchmark Models

export FAL_KEY=your_fal_key

# Text-to-image
evaly bench -m flux-schnell -m flux-dev -m flux-pro \
  -p "A cat sitting on a windowsill" --yes

# Image-to-image
evaly bench -m flux-kontext -m seedream-edit -m reve-edit \
  --inputs product.jpg -p "Place on a marble countertop" --yes

# Metrics only, no VLM judge
evaly bench -m flux-schnell -m flux-dev -p "A cat" --no-judge

5. Interactive Setup

evaly init   # Guided setup: use case, API keys, config file

CLI Commands

Command	Description
evaly init	Interactive setup wizard
evaly demo	Browse real benchmark showcases (no API key needed)
evaly bench	Generate, score, and report in one command
evaly eval	Score a single image without generation
evaly gate	CI/CD quality gate with pass/fail exit codes

Judges

Any VLM that can analyze images works as a judge:

evaly bench -m flux-schnell -p "A cat" -j gemini-2.5-flash            # Default
evaly bench -m flux-schnell -p "A cat" -j openai/gpt-5.2              # OpenAI
evaly bench -m flux-schnell -p "A cat" -j anthropic/claude-sonnet-4-6 # Anthropic
evaly bench -m flux-schnell -p "A cat" -j fal/gemini-2.5-flash        # Via fal.ai (single key)
evaly bench -m flux-schnell -p "A cat" -j ollama/qwen2.5-vl:7b        # Local

Consensus Mode

Use multiple judges for more reliable scores:

evaly bench -m flux-schnell -p "A cat" \
  --judges "gemini-2.5-flash,openai/gpt-5.2"

Two judges score in parallel. If they disagree, a third breaks the tie.

Optional Extras

pip install "evalytic[metrics]"  # CLIP Score + LPIPS + ArcFace (~2GB)
pip install "evalytic[all]"      # Everything

Configuration

Create evalytic.toml in your project root:

[keys]
fal = "your_fal_key"
gemini = "your_gemini_key"

[bench]
judge = "gemini-2.5-flash"
dimensions = ["visual_quality", "prompt_adherence"]
concurrency = 4

[bench.dimension_weights]
input_fidelity = 0.5
visual_quality = 0.1

Documentation

Full docs at docs.evalytic.ai

License

MIT