EvalForm

Declarative AI evaluation testing from a single YAML file.

EvalForm lets you test an AI or RAG application from one YAML file. It sends your test questions to your application, runs an evaluator, and fails a CI job when a quality rule is not met.

⚠️ Alpha Release

EvalForm is in active development. While the core architecture is stable:

• Provider packs have been validated against live APIs
• Configuration schema may evolve based on user feedback
• Docker images are published to GHCR
• Documentation is being actively expanded

We welcome early adopters! Please report issues and share feedback.

Included providers

• RAGAS for RAG metrics such as faithfulness
• DeepEval for LLM evaluation metrics
• Promptfoo for red-team probes
• Mock for deterministic tests without an API key

EvalForm runs providers in Docker containers. You install EvalForm itself, but you do not need to install RAGAS, DeepEval, Node.js, or Promptfoo locally.

Prerequisites

• Python 3.9-3.13 (Python 3.14+ has asyncio compatibility issues with RAGAS in local mode)
• Docker Desktop or Docker Engine (required for Docker mode with provider containers)
  • Optional: Use --local flag to run without Docker (faster, but requires compatible Python version)

Install

From PyPI

pip install evalform
# OR
python -m pip install evalform

# If using corporate pip config, install from public PyPI:
pip install evalform --index-url https://pypi.org/simple/

# Verify installation
evalform --version
#or 
python -m pip show evalform
evalform providers

Windows Users: If evalform command is not found:

Option 1 - Add to PATH (Recommended):

1. Search for "Environment Variables" in Windows
2. Edit your user PATH variable
3. Add: C:\Users\<YourUsername>\AppData\Roaming\Python\Python3XX\Scripts
4. Restart your terminal
5. Now you can use evalform directly

Option 2 - Use full path each time:

& "C:\Users\<YourUsername>\AppData\Roaming\Python\Python3XX\Scripts\evalform.exe" providers

Note: If you see cel-python errors, install it:

pip install cel-python

Docker Images

Important: Docker images are required for Docker mode execution.

Pull RAGAS Image (Required for RAG evaluation)

For Apple Silicon (M1/M2/M3) Mac users:

# Simple pull works natively
docker pull ghcr.io/vasundhra02/evalform-ragas:0.0.1

# Verify the image
docker images | grep evalform-ragas

For Windows and Intel Mac users:

# Pull with specific digest for platform compatibility
# Current digest: a6914c29eaf62f8395245323bdc7b3d3d7d270a0b0695b43f12bed3c430a708c
docker pull ghcr.io/vasundhra02/evalform-ragas:0.0.1@sha256:<digest>

# Tag it for evalform to find
docker tag ghcr.io/vasundhra02/evalform-ragas@sha256:<digest> ghcr.io/vasundhra02/evalform-ragas:0.0.1

# Verify the image
docker images | grep evalform-ragas

Example with current digest:

docker pull ghcr.io/vasundhra02/evalform-ragas:0.0.1@sha256:a6914c29eaf62f8395245323bdc7b3d3d7d270a0b0695b43f12bed3c430a708c
docker tag ghcr.io/vasundhra02/evalform-ragas@sha256:a6914c29eaf62f8395245323bdc7b3d3d7d270a0b0695b43f12bed3c430a708c ghcr.io/vasundhra02/evalform-ragas:0.0.1

Optional: Pull Other Provider Images

For Apple Silicon (M1/M2/M3) Mac users:

# DeepEval
docker pull ghcr.io/vasundhra02/evalform-deepeval:0.0.1

# Promptfoo
docker pull ghcr.io/vasundhra02/evalform-promptfoo:0.0.1

For Windows and Intel Mac users:

# DeepEval - check GitHub packages for latest digest
docker pull ghcr.io/vasundhra02/evalform-deepeval:0.0.1@sha256:<digest>
docker tag ghcr.io/vasundhra02/evalform-deepeval@sha256:<digest> ghcr.io/vasundhra02/evalform-deepeval:0.0.1

# Promptfoo - check GitHub packages for latest digest
docker pull ghcr.io/vasundhra02/evalform-promptfoo:0.0.1@sha256:<digest>
docker tag ghcr.io/vasundhra02/evalform-promptfoo@sha256:<digest> ghcr.io/vasundhra02/evalform-promptfoo:0.0.1

Note: Find the latest digests at https://github.com/Vasundhra02?tab=packages

Platform Note:

• Images were built on Apple Silicon (M1/M2/M3) and are linux/arm64 architecture
• Apple Silicon users: Images will run natively (no emulation needed)
• Windows and Intel Mac users: Docker will run them via emulation (x86_64 → arm64)
• This is why you need to use the specific digest - Docker can't auto-detect the right platform
• Performance may be slightly slower on Windows/Intel due to emulation, but functionality is identical
• Future: Multi-platform images (arm64 + amd64) will be published to eliminate this step

Create a starter configuration:

evalform init --kind mixed --name my-evaluation

This creates:

• evalform.yaml - Suite configuration
• evalform-cases.jsonl - Test cases

Now customize these files for your use case (see examples below).

Add credentials

Create .env in the same directory as your suite:

OpenAI

OPENAI_API_KEY=sk-your-api-key

Azure OpenAI

LLM_API_KEY=your-azure-api-key
LLM_URL=https://your-resource.openai.azure.com
AZURE_OPENAI_API_VERSION=2024-02-01

Important Notes:

• EvalForm's .env loader does NOT support variable substitution (e.g., ${VAR_NAME})
• Use literal values only: KEY=value format
• Never commit .env files to version control
• For Docker mode, ensure API keys are passed via execution.env_passthrough in your suite YAML

Azure OpenAI Configuration: The RAGAS provider checks for API keys in this order:

1. config.azure_api_key in the suite YAML
2. LLM_API_KEY environment variable
3. AZURE_OPENAI_API_KEY environment variable

You can either:

• Set the key directly in YAML: config.azure_api_key: your-key (not recommended for production)
• Use environment variables and pass them via env_passthrough: [LLM_API_KEY]

Customize the Suite Configuration

After running evalform init, you need to customize evalform.yaml for your specific use case.

Key Sections to Update:

1. Target Configuration

Update the target section to point to your API:

target:
  system: my-rag-app          # Your system name
  environment: local          # or staging, production
  runner:
    kind: http
    url: http://localhost:8000/chat    # Your API endpoint
    method: POST
    request_map:
      question: question      # Maps JSONL field to API request
    response_map:
      answer: answer          # Maps API response to evalform variable
      contexts: contexts      # For RAG systems

Important:

• request_map: Maps test case fields → API request body
• response_map: Maps API response fields → variables for metrics
• When using response_map, test cases only need input fields (e.g., question)

2. Metrics Configuration

Update the metrics section for your LLM provider:

For Azure OpenAI:

metrics:
  - id: faithfulness
    provider: ragas
    metric: faithfulness
    mode: score
    config:
      judge_model: gpt-4o-mini              # Your Azure deployment name
      azure_endpoint: https://your-resource.openai.azure.com
      azure_api_version: "2024-02-01"
      azure_api_key: your-key-here          # Or use env var
    map:
      question: question
      contexts: contexts
      answer: answer

For OpenAI:

metrics:
  - id: faithfulness
    provider: ragas
    metric: faithfulness
    mode: score
    config:
      judge_model: gpt-4o-mini
    map:
      question: question
      contexts: contexts
      answer: answer

3. Test Cases Format

Update evalform-cases.jsonl based on your response_map:

With response_map (recommended for live APIs):

{"id": "case-1", "question": "What is your return policy?"}
{"id": "case-2", "question": "Do you ship internationally?"}

The answer and contexts will come from your API response.

Without response_map (static test data):

{"id": "case-1", "question": "What is your return policy?", "contexts": ["Returns accepted within 30 days."], "answer": "You can return items within 30 days."}

4. Execution Mode

Choose between Docker or local mode:

execution:
  mode: docker                    # Recommended: reproducible, Python 3.14 compatible
  env_passthrough: [LLM_API_KEY]  # Pass environment variables to container

OR

execution:
  mode: local                     # Faster, but requires Python 3.9-3.13

Suite Examples

Example 1: RAG System with Live API (OpenAI)

version: 1
suite: support-bot-quality

target:
  system: support-bot
  environment: staging
  runner:
    kind: http
    url: https://staging.example.com/chat
    method: POST
    request_map:
      question: question
    response_map:
      answer: answer
      contexts: contexts

test_data:
  source: file
  path: ./evalform-cases.jsonl

metrics:
  - id: faithfulness
    provider: ragas
    metric: faithfulness
    mode: score
    config:
      judge_model: gpt-4o-mini
    map:
      question: question
      contexts: contexts
      answer: answer

policy:
  - name: faithfulness-floor
    when: "metric.id == 'faithfulness'"
    assert: "normalized.value >= 0.8"

execution:
  mode: docker
  env_passthrough: [OPENAI_API_KEY]

Example 2: RAG System with Live API (Azure OpenAI)

version: 1
suite: rag-chatbot-quality

target:
  system: rag-chatbot
  environment: local
  runner:
    kind: http
    url: http://localhost:8000/chat
    method: POST
    request_map:
      question: question
    response_map:
      answer: answer
      contexts: contexts

test_data:
  source: file
  path: ./evalform-cases.jsonl

metrics:
  - id: faithfulness
    provider: ragas
    metric: faithfulness
    mode: score
    config:
      judge_model: gpt-4o-mini              # Azure deployment name
      azure_endpoint: https://your-resource.openai.azure.com
      azure_api_version: "2024-02-01"
      azure_api_key: your-key-here          # Or use LLM_API_KEY env var
    map:
      question: question
      contexts: contexts
      answer: answer

policy:
  - name: faithfulness-floor
    when: "metric.id == 'faithfulness'"
    assert: "normalized.value >= 0.7"

execution:
  mode: docker
  env_passthrough: [LLM_API_KEY]

Test cases (evalform-cases.jsonl):

{"id": "case-1", "question": "What is your return policy?"}
{"id": "case-2", "question": "Do you ship internationally?"}

Test Cases

Each JSONL line is one test question:

{"id":"case-1","question":"What is your return policy?"}
{"id":"case-2","question":"Do you ship internationally?"}

Note: When using response_map to get answer and contexts from your API, you don't need to include them in the test cases. The test cases only need the input fields (e.g., question).

Change provider and metric to use DeepEval. For Promptfoo, use mode: probe and set target_endpoint, plugins, and num_probes in the metric config. The example suites in examples/ show each provider.

Run an evaluation

evalform plan --suite evalform.yaml   # Validate configuration (dry run)
evalform apply --suite evalform.yaml  # Run actual evaluation

plan checks the configuration without evaluator calls. apply runs your target and evaluator, prints scores, applies policies, and saves history in .evalform/.

Note: plan may show warnings about missing fields (like contexts) when using response_map. This is expected - those fields will be populated from the API response during apply.

For a smoke test without credentials:

evalform apply --suite examples/suite-mock.yaml --local --no-save

Exit codes are 0 for pass, 1 for a failed quality policy, and 2 for a configuration, provider, credential, or execution error.

Quick Troubleshooting

"Azure OpenAI requires an API key" error:

• Ensure LLM_API_KEY is set in .env file (no variable substitution like ${VAR})
• Add LLM_API_KEY to execution.env_passthrough list in your suite YAML
• OR set config.azure_api_key directly in the metric config (not recommended for production)

"no matching manifest for linux/amd64" error:

• Pull the image with the specific digest (see installation section above)
• Images are built for ARM64 but will run via emulation on x86_64

Python 3.14 asyncio errors with local mode:

• Use Docker mode instead: execution.mode: docker
• OR downgrade to Python 3.13 or earlier for local mode

"evalform: command not found":

• Add Python Scripts directory to PATH (see installation section)
• OR use full path to evalform.exe
• OR use python -m evalform instead

CI

Store the API key as a CI secret:

- run: pip install evalform
- run: evalform apply --suite evalform.yaml
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Persist .evalform/ between runs if you want baseline comparisons.

Add a provider

Providers are plug-ins described by YAML manifests, not hard-coded into EvalForm.

Interactive mode (recommended):

evalform provider create

Non-interactive mode:

evalform provider init trulens \
  --package trulens_eval \
  --image ghcr.io/your-org/evalform-trulens:0.1.0

This creates a provider manifest, dependency file, and fixtures. Edit the manifest to declare the provider version, Docker image, required environment variables, config fields, input mapping, and output-to-score mapping.

Validate and build it without changing EvalForm code:

evalform provider validate providers/trulens
evalform provider test providers/trulens      # Test with fixtures (no API key)
evalform provider doctor providers/trulens    # Pre-publish checks
evalform provider build providers/trulens --tag evalform/trulens:0.1.0

See docs/providers.md for the complete guide.

Known Limitations

Current Limitations

• Test data sources: Only file (JSONL) is supported. Live trace ingestion from observability platforms is planned.
• Target runners: Only HTTP POST is implemented. GraphQL, gRPC, and custom runners are planned.
• Storage backends: Only SQLite. Postgres support is planned for shared team baselines.
• Provider coverage: RAGAS, DeepEval, Promptfoo, and Mock are included. Community contributions for TruLens, Garak, LangSmith adapters are welcome.
• Windows support: Tested on Windows 11 with Docker Desktop. WSL2 backend recommended.
• Docker-in-Docker: If running EvalForm inside a container, bind mounts must be on a shared volume accessible to the host Docker daemon.

Workarounds

No Docker available?

evalform apply --suite evalform.yaml --local

Note: Results are stamped with execution_mode: local and depend on your installed libraries.

Baseline too stale?

baseline:
  strategy: rolling_window  # Average last N passing runs
  window: 5

Provider not available? Create a custom provider pack (see docs/providers.md) or open an issue requesting it.

Documentation

• Troubleshooting Guide - Common errors and solutions
• Provider Creation - Add new evaluation tools
• Examples - Sample suite configurations
• CHANGELOG - Version history
• CONTRIBUTING - Development guidelines

Roadmap

Beta (Q3 2024):

• Stabilize configuration schema
• Add more provider packs (TruLens, Garak)
• Postgres storage backend
• Web UI for result visualization

v1.0:

• Live trace ingestion
• GraphQL/gRPC target runners
• Hosted service option
• Performance optimizations for large test suites

See GitHub Issues for detailed planning.

Contributing

We welcome contributions! See CONTRIBUTING.md for:

• How to add provider packs (no Python required!)
• Development setup
• Testing guidelines
• Code of conduct

Quick wins for contributors:

• Add provider packs for your favorite eval tools
• Improve documentation
• Report bugs with minimal reproducers
• Share your suite configurations as examples

Support

• Issues: https://github.com/Vasundhra02/evalform/issues
• Discussions: https://github.com/Vasundhra02/evalform/discussions
• Email: [Your support email if available]

License

Apache-2.0 - See LICENSE for details.

Acknowledgments

Built with:

• CEL-Python for policy expressions
• RAGAS for RAG evaluation
• Promptfoo for red-teaming
• DeepEval for LLM evaluation