TestBench AI Service

An asynchronous REST API service that connects imbus TestBench to external LLM providers to support AI-driven Agents during test design and execution.

Features

• Multiple Agents: test case set reviews, description generation, and defect explanations, each configurable independently
• Pluggable LLM providers: ships with OpenAI, Azure OpenAI, and Anthropic support; implement a custom LLMClient to bring your own
• Automatic provider routing: model names starting with gpt-* and o-series (o1, o3, o4-mini, …) are routed to OpenAI, claude-* to Anthropic — regardless of the global provider setting
• Configurable prompts: YAML-based templates with Jinja2 variables and per-project overrides
• Per-project configuration: language, LLM provider, prompt variant, and enabled agents can be overridden per TestBench project
• Localization: built-in English and German translations for AI-generated output
• JWT authentication: every request is validated against the TestBench REST API using a JWT token; no separate credential management
• Async processing: Agents run as background tasks so the API responds immediately
• Swagger UI: interactive API docs at /docs
• SSL/TLS & reverse proxy support: optional HTTPS with mTLS and trusted-proxy headers

Installation

With pip (Python 3.10–3.14 required):

pip install testbench-ai-service

Standalone executable (no Python required): download the pre-built binary from the GitHub releases page.

Quick Start

1. Set your LLM API key

Create a .env file in the installation directory with the key for your chosen provider:

# .env
# OpenAI
OPENAI_API_KEY=your_openai_api_key

# Azure OpenAI
# AZURE_OPENAI_API_KEY=your_azure_openai_api_key

# Anthropic
# ANTHROPIC_API_KEY=your_anthropic_api_key

2. Initialize configuration

testbench-ai-service init

Open config.toml and point tb_server_url at your TestBench REST API:

[testbench-ai-service]
tb_server_url = "https://localhost:9443/api/"

[testbench-ai-service.llm_config]
# openai (default), azure_openai, anthropic, or custom
provider = "openai"

3. Start the service

testbench-ai-service start

The service runs at http://127.0.0.1:8010 by default. Open /docs for the interactive Swagger UI.

Documentation

Full documentation is available in the docs/ folder of the repository:

• Introduction
• Installation
• Quickstart
• Configuration
• Agents
• Prompts
• TestBench Integration
• CLI Reference

Contributing

Contributions are welcome. See CONTRIBUTING.md for setup instructions and guidelines.

Changelog

See CHANGELOG.md for release history.

License

Apache 2.0. See LICENSE for details.