Eval Agents (AgentLab)

A modular, composable framework for building, evaluating, and comparing LLM-powered agents.

You write the agent logic. The framework provides the LLMs, embeddings, and vector stores, and automatically runs the combinatorial experiments to find the best stack.

Installation

pip install eval-agents

(Note: While the package is published on PyPI as eval-agents, you import it as agentlab in your code).

The Magic of AgentLab

The beauty of this framework is that you only write your agent logic. You don't need to write LLM integrations or hardcode API keys. AgentLab provides a built-in catalogue of OpenAI, Anthropic, Google, and local models.

1. Write your agent (my_project.py)

from agentlab import agent

@agent(name="My Planner")
class PlannerAgent:
    def __init__(self, llm, embedding=None, vectorstore=None, reranker=None):
        # The framework will automatically inject the real implementations here!
        self.llm         = llm          
        self.embedding   = embedding    
        self.vectorstore = vectorstore  
        self.reranker    = reranker     

    def run(self, query: str) -> str:
        # Just use the injected LLM — you don't need to know if it's GPT-4o or Claude!
        return self.llm.generate(f"Plan this task: {query}")

2. Launch the UI (Coming Soon)

agentlab start --app my_project.py

3. Run Experiments!

The UI will automatically discover your @agent classes.

1. Select your agent: ☑ My Planner
2. Select LLMs to test: ☑ GPT-4o ☑ Claude Sonnet ☑ Gemini 2.5 Pro
3. Enter your API keys in the secure UI settings.
4. Click Run!

The framework will execute every combination and show you a dashboard comparing latency, token costs, and response quality.

Package Structure (Internal)

If you are contributing to AgentLab, the internal structure looks like this:

License

MIT