EvalForm

Declarative AI evaluation testing from a single YAML file.

EvalForm lets you test an AI or RAG application from one YAML file. It sends your test questions to your application, runs an evaluator, and fails a CI job when a quality rule is not met.

⚠️ Alpha Release

EvalForm is in active development. While the core architecture is stable:

• Provider packs have been validated against live APIs
• Configuration schema may evolve based on user feedback
• Docker images are published to GHCR
• Documentation is being actively expanded

We welcome early adopters! Please report issues and share feedback.

Included providers

• RAGAS for RAG metrics such as faithfulness
• DeepEval for LLM evaluation metrics
• Promptfoo for red-team probes
• Mock for deterministic tests without an API key

EvalForm runs providers in Docker containers. You install EvalForm itself, but you do not need to install RAGAS, DeepEval, Node.js, or Promptfoo locally.

Prerequisites

• Python 3.9 or later
• Docker Desktop or Docker Engine (for reproducible execution)
  • Optional: Use --local flag to run without Docker (faster, but results depend on local installs)

Install

pip install evalform
evalform --version
evalform providers

Note: If you see cel-python errors, install it:

pip install cel-python

Create a starter configuration:

evalform init --kind mixed --name my-evaluation

This creates a suite YAML file and a JSONL test-case file. Edit the generated files with your target URL, response fields, metrics, and quality thresholds.

Add credentials

Create .env in the same directory as your suite:

OPENAI_API_KEY=your-api-key

EvalForm passes only variables declared by the provider to its container. Never commit .env or share your key. Azure OpenAI users can configure the Azure endpoint, deployment, API version, and key in the metric config and environment.

Create a suite

Example RAGAS suite:

version: 1
suite: support-bot-quality

target:
  system: support-bot
  environment: staging
  runner:
    kind: http
    url: https://staging.example.com/chat
    method: POST
    request_map:
      question: question
    response_map:
      answer: answer
      contexts: contexts

test_data:
  source: file
  path: ./evalform-cases.jsonl

metrics:
  - id: faithfulness
    provider: ragas
    metric: faithfulness
    mode: score
    config:
      judge_model: gpt-4o-mini
    map:
      question: question
      contexts: contexts
      answer: answer

policy:
  - name: faithfulness-floor
    when: "metric.id == 'faithfulness'"
    assert: "normalized.value >= 0.8"

execution:
  mode: docker
  docker:
    env_passthrough: [OPENAI_API_KEY]

Each JSONL line is one test question:

{"id":"case-1","question":"What is your return policy?"}
{"id":"case-2","question":"Do you ship internationally?"}

Change provider and metric to use DeepEval. For Promptfoo, use mode: probe and set target_endpoint, plugins, and num_probes in the metric config. The example suites in examples/ show each provider.

Run an evaluation

evalform plan --suite evalform.yaml
evalform apply --suite evalform.yaml

plan checks the configuration without evaluator calls. apply runs your target and evaluator, prints scores, applies policies, and saves history in .evalform/.

For a smoke test without credentials:

evalform apply --suite examples/suite-mock.yaml --local --no-save

Exit codes are 0 for pass, 1 for a failed quality policy, and 2 for a configuration, provider, credential, or execution error.

CI

Store the API key as a CI secret:

- run: pip install evalform
- run: evalform apply --suite evalform.yaml
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Persist .evalform/ between runs if you want baseline comparisons.

Add a provider

Providers are plug-ins described by YAML manifests, not hard-coded into EvalForm.

Interactive mode (recommended):

evalform provider create

Non-interactive mode:

evalform provider init trulens \
  --package trulens_eval \
  --image ghcr.io/your-org/evalform-trulens:0.1.0

This creates a provider manifest, dependency file, and fixtures. Edit the manifest to declare the provider version, Docker image, required environment variables, config fields, input mapping, and output-to-score mapping.

Validate and build it without changing EvalForm code:

evalform provider validate providers/trulens
evalform provider test providers/trulens      # Test with fixtures (no API key)
evalform provider doctor providers/trulens    # Pre-publish checks
evalform provider build providers/trulens --tag evalform/trulens:0.1.0

See docs/providers.md for the complete guide.

Known Limitations

Current Limitations

• Test data sources: Only file (JSONL) is supported. Live trace ingestion from observability platforms is planned.
• Target runners: Only HTTP POST is implemented. GraphQL, gRPC, and custom runners are planned.
• Storage backends: Only SQLite. Postgres support is planned for shared team baselines.
• Provider coverage: RAGAS, DeepEval, Promptfoo, and Mock are included. Community contributions for TruLens, Garak, LangSmith adapters are welcome.
• Windows support: Tested on Windows 11 with Docker Desktop. WSL2 backend recommended.
• Docker-in-Docker: If running EvalForm inside a container, bind mounts must be on a shared volume accessible to the host Docker daemon.

Workarounds

No Docker available?

evalform apply --suite evalform.yaml --local

Note: Results are stamped with execution_mode: local and depend on your installed libraries.

Baseline too stale?

baseline:
  strategy: rolling_window  # Average last N passing runs
  window: 5

Provider not available? Create a custom provider pack (see docs/providers.md) or open an issue requesting it.

Documentation

• Troubleshooting Guide - Common errors and solutions
• Provider Creation - Add new evaluation tools
• Examples - Sample suite configurations
• CHANGELOG - Version history
• CONTRIBUTING - Development guidelines

Roadmap

Beta (Q3 2024):

• Stabilize configuration schema
• Add more provider packs (TruLens, Garak)
• Postgres storage backend
• Web UI for result visualization

v1.0:

• Live trace ingestion
• GraphQL/gRPC target runners
• Hosted service option
• Performance optimizations for large test suites

See GitHub Issues for detailed planning.

Contributing

We welcome contributions! See CONTRIBUTING.md for:

• How to add provider packs (no Python required!)
• Development setup
• Testing guidelines
• Code of conduct

Quick wins for contributors:

• Add provider packs for your favorite eval tools
• Improve documentation
• Report bugs with minimal reproducers
• Share your suite configurations as examples

Support

• Issues: https://github.com/Vasundhra02/evalform/issues
• Discussions: https://github.com/Vasundhra02/evalform/discussions
• Email: [Your support email if available]

License

Apache-2.0 - See LICENSE for details.

Acknowledgments

Built with:

• CEL-Python for policy expressions
• RAGAS for RAG evaluation
• Promptfoo for red-teaming
• DeepEval for LLM evaluation