Eval Agents

A modular, composable framework for building, evaluating, and comparing LLM-powered agents.

You write the agent logic. The framework provides the LLMs, embeddings, vector stores, and rerankers — and automatically runs combinatorial experiments to find the best stack.

Installation

pip install eval-agents

Or with optional provider extras:

pip install eval-agents[llm]      # OpenAI, Anthropic, Google SDKs
pip install eval-agents[rerank]   # Cohere + CrossEncoder rerankers
pip install eval-agents[dev]      # Development tools

Quick Start

from agentlab.agents import agent
from agentlab.llm import get_llm
from agentlab.embedding import get_embedding
from agentlab.retriever import get_retriever
from agentlab.reranker import get_reranker
from agentlab.execution import run_experiment

# 1. Define your agent using the @agent decorator
@agent(name="my_researcher")
class MyResearcher:
    def __init__(self, llm, embedding=None, vectorstore=None, reranker=None):
        self.llm         = llm
        self.embedding   = embedding
        self.vectorstore = vectorstore
        self.reranker    = reranker

    def run(self, query: str) -> str:
        return self.llm.generate(f"Research: {query}")

# 2. Build components
llm       = get_llm(provider="openai", model="gpt-4o")
embedding = get_embedding(provider="openai", model="text-embedding-3-small")
retriever = get_retriever(provider="faiss")
reranker  = get_reranker(provider="cohere")

# 3. Launch the UI
# agentlab start --app my_project.py

# 4. Run an experiment
result = run_experiment({
    "agents":     ["my_researcher"],
    "llms":       [{"provider": "openai",  "model": "gpt-4o",                "temperature": 0.2}],
    "embeddings": [{"provider": "openai",  "model": "text-embedding-3-small"}],
    "retrievers": [{"provider": "faiss"}],
    "mode": "Sequential"
})

Package Structure

Eval Agents is organized like scoped packages — each sub-package has one clear responsibility:

Run Experiments via UI

The AgentLab UI allows you to:

1. Register agents using the @agent decorator
2. Select LLMs to test: ☑ GPT-4o ☑ Claude Sonnet ☑ Gemini 2.5 Pro
3. Select Embeddings: ☑ text-embedding-3-small ☑ all-MiniLM-L6-v2
4. Enter API keys in the secure settings panel
5. Click Run — the engine runs every combination and shows a comparison dashboard

agentlab start --app my_project.py

Environment Variables

Create a .env file in your project root:

OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AI...
COHERE_API_KEY=...
VOYAGE_API_KEY=...

License

MIT