Skip to main content

AgentCI Client Config

Define evaluations and framework configurations for AI agent applications using simple TOML files.

📚 Full Documentation | 🚀 Getting Started | 📖 TOML Schema Guides

What is This?

AgentCI Client Config provides a TOML-based configuration format for:

  • Evaluations: Test cases for AI agents and tools with support for accuracy, performance, consistency, and safety testing
  • Framework Configurations: Patterns for discovering agents and tools in popular AI frameworks (LangChain, LlamaIndex, Pydantic AI, OpenAI, Google ADK, Agno)

Quick Example

Create evaluation configs in .agentci/evals/:

# .agentci/evals/test_accuracy.toml
[eval]
description = "Test that the agent responds with correct information"
type = "accuracy"

[eval.targets]
agents = ["my_agent"]

[[eval.cases]]
prompt = "What is the capital of France?"
expected.exact = "Paris"

Create framework configs in .agentci/frameworks/:

# .agentci/frameworks/my_framework.toml
[framework]
name = "my-framework"
dependencies = ["my-framework"]

[[agents]]
path = "my_framework.Agent"
args.model = "llm"
args.prompt = "system_prompt"
execution.method = "run"
execution.args.prompt = "user_input"

Installation

pip install agentci-client-config

Documentation

For complete TOML schema documentation and guides:

Features

Evaluations

  • Six evaluation types: accuracy, performance, consistency, safety, llm, custom
  • Flexible matching: exact, contains, regex, semantic similarity
  • Schema validation: Validate structured JSON outputs
  • Tool call validation: Verify correct tool usage
  • Multiple iterations: Run tests multiple times for consistency

Frameworks

  • Built-in support: LangChain, LlamaIndex, Pydantic AI, OpenAI Agents, Google ADK, Agno
  • Custom frameworks: Define your own discovery patterns
  • Agent discovery: Map framework parameters to standard fields
  • Tool discovery: Configure tool types (decorator, function, class, constructor)
  • Execution config: Define how to run agents and tools

Directory Structure

your-project/
├── .agentci/
│   ├── evals/              # Evaluation configurations
│   │   ├── accuracy.toml
│   │   ├── performance.toml
│   │   └── safety.toml
│   └── frameworks/         # Framework configurations (optional)
│       └── custom.toml
├── src/
└── tests/

License

MIT

Metadata

Release files for agentci-client-config 0.1.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agentci-client-config 0.1.3
File Size Uploaded Provenance Actions
agentci_client_config-0.1.3.tar.gz 105.6 kB Without provenance Details for agentci_client_config-0.1.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agentci-client-config 0.1.3
File Interpreter ABI Platform Actions
agentci_client_config-0.1.3-py3-none-any.whl Python 3 none any Details for agentci_client_config-0.1.3-py3-none-any.whl

Total release size: 122.4 kB

Release files / agentci_client_config-0.1.3.tar.gz

Download URL agentci_client_config-0.1.3.tar.gz
Size 105.6 kB
Tags Source
SHA-256 checksum
How to use checksums
7ede686c7cc6e4545c8a86b8f60c679cce836f5b47cf0516ef6cf7ca2b01394a
BLAKE2b-256 checksum
How to use checksums
9e937d88b9c8f8c807d815a8f9aae4711e67b12af02b069080a578ba8c15e677
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.5

Release files / agentci_client_config-0.1.3-py3-none-any.whl

Download URL agentci_client_config-0.1.3-py3-none-any.whl
Size 16.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3f5201e9cbfce69d656eaf928d5801be7b210d8e8f27a253a689a185568d46b4
BLAKE2b-256 checksum
How to use checksums
49388f05efce1165cb13c2a12080f7c43046e6667224f2d112472ce5438bea16
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.5

Security for agentci-client-config 0.1.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page