ASQI Engineer

ASQI (AI Solutions Quality Index) Engineer helps teams test and evaluate AI systems. It runs containerized test packages, automates scoring, and provides durable execution workflows.

The project focuses first on chatbot testing and supports extensions for other AI system types. Resaro welcomes contributions of test packages, score cards, and schemas.

Table of Contents

• AI System Testing
• Quick Start
• Documentation
• Key Highlights
• Contributing & development
• License

Key Features

Modular Test Execution

• Durable execution: DBOS-powered fault tolerance with automatic retry and recovery
• Concurrent testing: Parallel test execution with configurable concurrency limits
• Container isolation: Each test runs in isolated Docker containers for consistency and reproducibility

Flexible Scenario-based Testing

• Core schema definition: Specifies the underlying contract between test packages and users running tests, enabling an extensible approach to scale to new use cases and test modules
• Multi-system orchestration: Tests can coordinate multiple AI systems (target, simulator, evaluator) in complex workflows
• Flexible configuration: Test packages specify input systems and parameters that can be customised for individual use cases

Dataset Support and Data Generation

• Input datasets: Feed evaluation datasets, source documents, or training data to test containers
• Dataset registry: Centralized dataset definitions with reusable configurations across test suites
• Multiple formats: Support for HuggingFace datasets, PDF documents, and text files
• Column mapping: Align dataset fields with container expectations for seamless integration
• Synthetic data generation: Generate training data, augment datasets, or create RAG question-answer pairs
• Output datasets: Containers can produce datasets as outputs for data pipeline workflows

Automated Assessment

• Structured reporting: JSON output with detailed metrics and assessment outcomes
• Configurable score cards: Define custom evaluation criteria with flexible assessment conditions
• Metric expressions: Combine multiple metrics using mathematical operations (+, -, *, /), comparison operators (>, >=, <, <=, ==, !=), boolean logic (and, or, not), conditional expressions (if-else), and functions (min, max, avg, abs, round, pow) for sophisticated composite scoring including hard gates patterns
• Technical reports: Enable test containers to generate html and pdf reports that provide detailed analysis and evidence for quality indicator assessments

Developer Experience

• Type-safe configuration: Pydantic schemas with JSON Schema generation for IDE support
• Rich CLI interface: Typer-based commands with comprehensive help and validation
• Real-time feedback: Live progress reporting with structured logging and tracing

AI System Testing

ASQI Engineer supports comprehensive testing across multiple AI system types including llm_api, rag_api, image_generation_api, image_editing_api, and vlm_api (vision-language models). This enables testing of traditional LLM APIs, Retrieval-Augmented Generation (RAG) systems with contextual retrieval capabilities, image generation and editing models, and multimodal vision-language systems. We have also open-sourced a draft ASQI score card for customer chatbots that provides mappings between technical metrics and business-relevant assessment criteria.

LLM Test Containers

• Garak: Security vulnerability assessment with 40+ attack vectors and probes
• DeepTeam: Red teaming library for adversarial robustness testing
• TrustLLM: Comprehensive framework and benchmarks to evaluate trustworthiness of LLM systems
• Inspect Evals: Comprehensive evaluation suite with 80+ tasks across cybersecurity, mathematics, reasoning, knowledge, bias, and safety domains
• LLMPerf: Token-level performance benchmarking for latency, throughput, and request metrics
• Resaro Chatbot Simulator: Persona and scenario based conversational testing with multi-turn dialogue simulation

The supported system types use OpenAI-compatible API interfaces, or in the case of rag_api, a superset of it. Through LiteLLM integration, ASQI Engineer provides unified access to 100+ LLM providers including OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, and custom endpoints. RAG systems additionally require responses with contextual citations for retrieval-augmented evaluation. This standardisation enables test containers to work seamlessly across different AI providers while supporting complex multi-system test scenarios (e.g., using different models for simulation, evaluation, and target testing).

Quick Start

Get started with ASQI Engineer in 3 simple steps:

Requirements

• Python 3.12+ is required
• Docker for running test containers (default), or a Kubernetes cluster (see Container Backend below)

  Note: If you are facing issues detecting your Docker daemon, you might need to set the DOCKER_HOST environment variable in your .env file. See .env for details.

Container Backend

ASQI Engineer supports two container backends, selected at startup via the RUN_BACKEND environment variable:

When using RUN_BACKEND=k8s, set K8S_NAMESPACE to the namespace where Jobs should be created (default: default). Apply the bundled RBAC manifest first:

kubectl apply -f "$(python -c 'from importlib.resources import files; print(files("asqi").joinpath("k8s/rbac.yaml"))')"

The backend is transparent to workflows — all five protocol methods (run, shutdown, check_images, pull_images, extract_manifest) behave identically from the caller's perspective regardless of which backend is active.

1. Install the package:

pip install asqi-engineer

2. Run the setup script:

curl -sSL https://raw.githubusercontent.com/asqi-engineer/asqi-engineer/main/setup.sh | bash

This downloads all required configuration files and creates a .env template.

3. Configure and run:

# Start the services and run your first test:
docker compose up -d
asqi execute-tests -t config/suites/demo_test.yaml -s config/systems/demo_systems.yaml

# Or generate synthetic data (if you have data generation containers):
asqi generate-dataset -t config/generation/suite.yaml -s config/systems/demo_systems.yaml -d config/datasets/registry.yaml

This short flow should download a demo test container and generate the test results in output.json. Now, to actually test your AI system, configure the .env file and try out the other test packages in: https://www.asqi.ai/quickstart.html

Documentation

Detailed documentation lives on the project docs site — use the links below to jump to the full guides and examples:

• Quickstart (installation & environment): https://www.asqi.ai/quickstart.html
• Library usage & workflow customization: docs/library.md
• CLI & usage reference: https://www.asqi.ai/cli.html
• Configuration & environment variables: https://www.asqi.ai/configuration.html
• Dataset support & data generation: docs/datasets.md
• Test container examples & how-to: https://www.asqi.ai/examples.html
• LLM test containers overview (Garak, DeepTeam, TrustLLM, Inspect Evals, LLMPerf, Chatbot Simulator, Resaro Judge): https://www.asqi.ai/llm-test-containers.html
• Score cards & evaluation: https://www.asqi.ai/examples.html#score-cards
• Developer guide & architecture: https://www.asqi.ai/architecture.html
• Creating custom test containers: https://www.asqi.ai/custom-test-containers.html

If a link is missing or the page content is unclear, please open an issue: https://github.com/asqi-engineer/asqi-engineer/issues

Key Highlights

• Durable, DBOS-backed execution with retries and recovery
• Containerized test packages for isolation and reproducibility
• Extensible test-suite and score-card model for automated assessment
• Pydantic-based schemas and rich CLI (Typer) for developer ergonomics

Contributing & development

We keep contributor-facing documentation split into two dedicated documents so each file stays concise and actionable.

Quick actions:

• To see how to contribute (PR process, templates, commit guidance), open CONTRIBUTING.md.
• To get your dev environment ready and run tests locally (venv, uv commands, and devcontainer), open DEVELOPMENT.md.
• Example configs and test containers live under config/ and test_containers/ respectively.

If you're unsure where to start, read CONTRIBUTING.md first for the workflow and then follow the setup steps in DEVELOPMENT.md to run the test suite locally.

License

Apache 2.0 © Resaro