agents-builder
agents-builder is a small Python framework for building retrieval-aware agents on top of LangGraph, LangChain, and MCP tools.
The package gives you a few focused primitives:
- A base
Agentthat owns graph construction, LLM factories, and MCP tool loading. - Config-driven runtime objects via Pydantic settings models.
- Reusable
Prompt,LLMNode,ToolBasedNode, andRouterNodebuilding blocks. - An agent evaluation API for running scenarios, scoring trajectories, and annotating traces.
- An agent evaluation API for running scenarios, scoring trajectories, and annotating traces.
- Lightweight support for multiple LLM backends such as Ollama and OpenRouter.
The design is intentionally narrow: compose agents from small graph nodes, keep configuration explicit, and avoid framework-heavy abstractions that hide behavior.
What This Project Contains
Core package layout:
src/agents_builder/
├── __init__.py # Base Agent
├── constants.py # Shared typing aliases and config path
├── exceptions.py # Domain-specific exceptions
├── mixins.py # Config loading helpers
├── settings.py # Pydantic settings models
├── utils.py # Dynamic loading, MCP caching, message helpers
├── eval/
│ ├── __init__.py # AgentEvalRunner, TraceAnnotator, assertions, scoring
│ ├── annotator.py # Phoenix trace annotation integration
│ ├── assertion.py # Reusable trajectory assertions
│ └── trajectory.py # EvalScenario and AgentTrajectory models
├── eval/
│ ├── __init__.py # AgentEvalRunner, TraceAnnotator, assertions, scoring
│ ├── annotator.py # Phoenix trace annotation integration
│ ├── assertion.py # Reusable trajectory assertions
│ └── trajectory.py # EvalScenario and AgentTrajectory models
├── llm/
│ ├── __init__.py
│ ├── factory.py # LLMFactory and role-based model lookup
│ └── llm.py # OllamaLLM / OpenRouterLLM implementations
└── langgraph/
├── __init__.py # Prompt, Node, LLMNode, DeletionStrategy
├── nodes.py # Ready-made retrieval/answering node types
└── states.py # Typed graph state
Tests are split by intent:
tests/unit: isolated behavior for factories, nodes, mixins, utils, and errors.tests/integration: config loading and caching behavior.tests/e2e: a full retrieval-answer graph wired together end to end.
Installation
This project uses uv and Python 3.12.
uv sync --group dev --all-extras
Common local commands:
make lint
make type-check
make test
make test-cov
make ci
Core Concepts
1. Agent
Subclass agents_builder.Agent and implement build_graph().
The base class already handles:
- config storage
- lazy graph compilation
- LLM factory initialization
- MCP tool loading with caching
- graph export via
push()
2. Settings-first runtime objects
Most runtime classes follow the same pattern:
- define a Pydantic settings model
- inherit from
FromConfigMixin[...] - construct the runtime object from validated config
This keeps runtime behavior strongly tied to validated configuration instead of ad hoc dictionaries.
3. Role-based LLM selection
LLMFactory maps roles like query, grader, or answer to specific model configs.
That makes it easy to:
- use different models for different steps
- swap providers without changing graph code
- keep model selection in config rather than inside node logic
4. LangGraph node building blocks
The package exposes low-level but reusable graph parts:
Prompt: render a system prompt from stateSchemaBasedPrompt: prompt plus structured output schemaLLMNode: invoke an LLM against rendered messagesToolBasedNode: base for MCP-backed tool nodesRouterNode: return route keys for conditional edgesDeletionStrategy: trim message history in a controlled way
5. Agent evaluation API
Use agents_builder.eval to run repeatable evaluation scenarios against a compiled agent graph.
The core flow is:
- define an
EvalScenariowith the user query and metadata - run the graph with
AgentEvalRunner - convert graph messages into an
AgentTrajectory - score the trajectory with
SuccessAssertionandEfficiencyAssertionchecks - emit trace annotations through a
TraceAnnotator
from agents_builder.eval import (
AgentTrajectory,
EfficiencyAssertion,
AgentEvalRunner,
EvalScenario,
SuccessAssertion,
TraceAnnotator,
TrajectoryScorer,
)
class RequiresAnswer(SuccessAssertion):
def check(self, trajectory: AgentTrajectory) -> bool:
passed = bool(trajectory.steps) and bool(trajectory.steps[-1].action.content)
self.logger.annotate(
annotation_name="answered",
annotator_kind="CODE",
label="pass" if passed else "fail",
score=1.0 if passed else 0.0,
explanation=trajectory.serialize(),
)
if not passed:
raise AssertionError("agent did not produce a final answer")
return passed
class LimitsStepCount(EfficiencyAssertion):
def __init__(self, logger: TraceAnnotator, max_steps: int) -> None:
super().__init__(logger)
self.max_steps = max_steps
def check(self, trajectory: AgentTrajectory) -> bool:
passed = len(trajectory.steps) <= self.max_steps
self.logger.annotate(
annotation_name="step_count",
annotator_kind="CODE",
label="pass" if passed else "fail",
score=1.0 if passed else 0.0,
metadata={"max_steps": self.max_steps, "actual_steps": len(trajectory.steps)},
)
return passed
scenario = EvalScenario(
name="contract lookup",
query="Find the renewal terms",
category="retrieval",
tier="smoke",
metadata={"dataset": "contracts"},
)
logger = ...
runner = AgentEvalRunner(graph=agent.graph, logger=logger)
scorer = TrajectoryScorer().success(RequiresAnswer(logger)).expect(LimitsStepCount(logger, max_steps=3))
trajectory = await runner.run(scenario, scorer)
For LLM-as-judge checks, use LLMJudgeAssertion from agents_builder.eval.assertion. Its prompt must contain exactly {query} and {trajectory} placeholders, and each criterion is emitted as an LLM trace annotation.
Phoenix-backed annotation is available through PhoenixTraceAnnotator with PhoenixTraceAnnotatorSettings:
from agents_builder.eval.annotator import PhoenixTraceAnnotator
from agents_builder.settings import PhoenixTraceAnnotatorSettings
logger = PhoenixTraceAnnotator(
PhoenixTraceAnnotatorSettings(
module_path="agents_builder.eval.annotator.PhoenixTraceAnnotator",
base_url="http://localhost:6006",
project_name="agents-builder",
)
)
Configuration
Configuration is built around Pydantic settings models in src/agents_builder/settings.py.
Important settings types:
AgentSettingsLLMFactorySettingsOllamaLLMSettingsOpenRouterLLMSettingsMCPServerSettingsTraceAnnotatorSettingsPhoenixTraceAnnotatorSettingsTraceAnnotatorSettingsPhoenixTraceAnnotatorSettings
The package-level default YAML path is:
/config/config.yaml
You can also construct objects directly from Python with .from_config(...) or load from YAML with .from_yaml(...).
Development Workflow
Quality checks already configured in the project:
rufffor linting and formattingtyfor type checkingpytestfor unit, integration, and e2e testsbanditfor security checks
Recommended local loop:
make lint
make type-check
make test
Before merging broader changes:
make ci
Design Principles
This repository works best when changes stay aligned with a few constraints:
- Prefer explicit graph nodes over magic orchestration layers.
- Keep configuration typed and validated.
- Use small runtime classes with narrow responsibilities.
- Let tests describe behavior at the node and graph level.
If you are contributing code or using AI coding agents in this repository, read AGENTS.md next.
Metadata
Release files for agents-builder 4.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agents_builder-4.0.0.tar.gz | 18.3 kB | Without provenance | Details for agents_builder-4.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agents_builder-4.0.0-py3-none-any.whl | Python 3 | none | any | Details for agents_builder-4.0.0-py3-none-any.whl |
Total release size: 41.9 kB
Release files / agents_builder-4.0.0.tar.gz
| Download URL | agents_builder-4.0.0.tar.gz |
|---|---|
| Size | 18.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9f9fb3b7a246d52605af0a482381237ff00ba7534b62385da577f95f3e8015f8
|
|
BLAKE2b-256 checksum How to use checksums |
6ac962b00655a9af98001f638616bbc8f0f7c8288794b4fd658e11279ed892e3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.16 {"installer":{"name":"uv","version":"0.11.16","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / agents_builder-4.0.0-py3-none-any.whl
| Download URL | agents_builder-4.0.0-py3-none-any.whl |
|---|---|
| Size | 23.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
fb0c784f53c5757f5c6cb0d89c8aaff8f84c78cfd21929d5d0b58655f1a9df34
|
|
BLAKE2b-256 checksum How to use checksums |
c664f9d2db1e6a090f6c18b6812f7b9c3b1fa0568c0422b431f9ae1a41f04ba0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.11.16 {"installer":{"name":"uv","version":"0.11.16","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for agents-builder 4.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.