AgentCI Client Config
Define evaluations and framework configurations for AI agent applications using simple TOML files.
📚 Full Documentation | 🚀 Getting Started | 📖 TOML Schema Guides
What is This?
AgentCI Client Config provides a TOML-based configuration format for:
- Evaluations: Test cases for AI agents and tools with support for accuracy, performance, consistency, and safety testing
- Framework Configurations: Patterns for discovering agents and tools in popular AI frameworks (LangChain, LlamaIndex, Pydantic AI, OpenAI, Google ADK, Agno)
Quick Example
Create evaluation configs in .agentci/evals/:
# .agentci/evals/test_accuracy.toml
[eval]
description = "Test that the agent responds with correct information"
type = "accuracy"
[eval.targets]
agents = ["my_agent"]
[[eval.cases]]
prompt = "What is the capital of France?"
expected.exact = "Paris"
Create framework configs in .agentci/frameworks/:
# .agentci/frameworks/my_framework.toml
[framework]
name = "my-framework"
dependencies = ["my-framework"]
[[agents]]
path = "my_framework.Agent"
args.model = "llm"
args.prompt = "system_prompt"
execution.method = "run"
execution.args.prompt = "user_input"
Installation
pip install agentci-client-config
Documentation
For complete TOML schema documentation and guides:
- Getting Started - Project setup and first configs
- Evaluation Schema - Complete guide to evaluation TOML format
- Framework Schema - Complete guide to framework TOML format
- Python API - Optional programmatic usage
Features
Evaluations
- Six evaluation types: accuracy, performance, consistency, safety, llm, custom
- Flexible matching: exact, contains, regex, semantic similarity
- Schema validation: Validate structured JSON outputs
- Tool call validation: Verify correct tool usage
- Multiple iterations: Run tests multiple times for consistency
Frameworks
- Built-in support: LangChain, LlamaIndex, Pydantic AI, OpenAI Agents, Google ADK, Agno
- Custom frameworks: Define your own discovery patterns
- Agent discovery: Map framework parameters to standard fields
- Tool discovery: Configure tool types (decorator, function, class, constructor)
- Execution config: Define how to run agents and tools
Directory Structure
your-project/
├── .agentci/
│ ├── evals/ # Evaluation configurations
│ │ ├── accuracy.toml
│ │ ├── performance.toml
│ │ └── safety.toml
│ └── frameworks/ # Framework configurations (optional)
│ └── custom.toml
├── src/
└── tests/
License
MIT
Metadata
Release files for agentci-client-config 0.1.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agentci_client_config-0.1.3.tar.gz | 105.6 kB | Without provenance | Details for agentci_client_config-0.1.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agentci_client_config-0.1.3-py3-none-any.whl | Python 3 | none | any | Details for agentci_client_config-0.1.3-py3-none-any.whl |
Total release size: 122.4 kB
Release files / agentci_client_config-0.1.3.tar.gz
| Download URL | agentci_client_config-0.1.3.tar.gz |
|---|---|
| Size | 105.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7ede686c7cc6e4545c8a86b8f60c679cce836f5b47cf0516ef6cf7ca2b01394a
|
|
BLAKE2b-256 checksum How to use checksums |
9e937d88b9c8f8c807d815a8f9aae4711e67b12af02b069080a578ba8c15e677
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.5
|
Release files / agentci_client_config-0.1.3-py3-none-any.whl
| Download URL | agentci_client_config-0.1.3-py3-none-any.whl |
|---|---|
| Size | 16.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3f5201e9cbfce69d656eaf928d5801be7b210d8e8f27a253a689a185568d46b4
|
|
BLAKE2b-256 checksum How to use checksums |
49388f05efce1165cb13c2a12080f7c43046e6667224f2d112472ce5438bea16
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.5
|
Security for agentci-client-config 0.1.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.