Skip to main content

agents-builder

agents-builder is a small Python framework for building retrieval-aware agents on top of LangGraph, LangChain, and MCP tools.

The package gives you a few focused primitives:

  • A base Agent that owns graph construction, LLM factories, and MCP tool loading.
  • Config-driven runtime objects via Pydantic settings models.
  • Reusable Prompt, LLMNode, ToolBasedNode, and RouterNode building blocks.
  • An agent evaluation API for running scenarios, scoring trajectories, and annotating traces.
  • An agent evaluation API for running scenarios, scoring trajectories, and annotating traces.
  • Lightweight support for multiple LLM backends such as Ollama and OpenRouter.

The design is intentionally narrow: compose agents from small graph nodes, keep configuration explicit, and avoid framework-heavy abstractions that hide behavior.

What This Project Contains

Core package layout:

src/agents_builder/
├── __init__.py          # Base Agent
├── constants.py         # Shared typing aliases and config path
├── exceptions.py        # Domain-specific exceptions
├── mixins.py            # Config loading helpers
├── settings.py          # Pydantic settings models
├── utils.py             # Dynamic loading, MCP caching, message helpers
├── eval/
│   ├── __init__.py      # AgentEvalRunner, TraceAnnotator, assertions, scoring
│   ├── annotator.py     # Phoenix trace annotation integration
│   ├── assertion.py     # Reusable trajectory assertions
│   └── trajectory.py    # EvalScenario and AgentTrajectory models
├── eval/
│   ├── __init__.py      # AgentEvalRunner, TraceAnnotator, assertions, scoring
│   ├── annotator.py     # Phoenix trace annotation integration
│   ├── assertion.py     # Reusable trajectory assertions
│   └── trajectory.py    # EvalScenario and AgentTrajectory models
├── llm/
│   ├── __init__.py
│   ├── factory.py       # LLMFactory and role-based model lookup
│   └── llm.py           # OllamaLLM / OpenRouterLLM implementations
└── langgraph/
    ├── __init__.py      # Prompt, Node, LLMNode, DeletionStrategy
    ├── nodes.py         # Ready-made retrieval/answering node types
    └── states.py        # Typed graph state

Tests are split by intent:

  • tests/unit: isolated behavior for factories, nodes, mixins, utils, and errors.
  • tests/integration: config loading and caching behavior.
  • tests/e2e: a full retrieval-answer graph wired together end to end.

Installation

This project uses uv and Python 3.12.

uv sync --group dev --all-extras

Common local commands:

make lint
make type-check
make test
make test-cov
make ci

Core Concepts

1. Agent

Subclass agents_builder.Agent and implement build_graph().

The base class already handles:

  • config storage
  • lazy graph compilation
  • LLM factory initialization
  • MCP tool loading with caching
  • graph export via push()

2. Settings-first runtime objects

Most runtime classes follow the same pattern:

  • define a Pydantic settings model
  • inherit from FromConfigMixin[...]
  • construct the runtime object from validated config

This keeps runtime behavior strongly tied to validated configuration instead of ad hoc dictionaries.

3. Role-based LLM selection

LLMFactory maps roles like query, grader, or answer to specific model configs.

That makes it easy to:

  • use different models for different steps
  • swap providers without changing graph code
  • keep model selection in config rather than inside node logic

4. LangGraph node building blocks

The package exposes low-level but reusable graph parts:

  • Prompt: render a system prompt from state
  • SchemaBasedPrompt: prompt plus structured output schema
  • LLMNode: invoke an LLM against rendered messages
  • ToolBasedNode: base for MCP-backed tool nodes
  • RouterNode: return route keys for conditional edges
  • DeletionStrategy: trim message history in a controlled way

5. Agent evaluation API

Use agents_builder.eval to run repeatable evaluation scenarios against a compiled agent graph.

The core flow is:

  • define an EvalScenario with the user query and metadata
  • run the graph with AgentEvalRunner
  • convert graph messages into an AgentTrajectory
  • score the trajectory with SuccessAssertion and EfficiencyAssertion checks
  • emit trace annotations through a TraceAnnotator
from agents_builder.eval import (
    AgentTrajectory,
    EfficiencyAssertion,
    AgentEvalRunner,
    EvalScenario,
    SuccessAssertion,
    TraceAnnotator,
    TrajectoryScorer,
)


class RequiresAnswer(SuccessAssertion):
    def check(self, trajectory: AgentTrajectory) -> bool:
        passed = bool(trajectory.steps) and bool(trajectory.steps[-1].action.content)
        self.logger.annotate(
            annotation_name="answered",
            annotator_kind="CODE",
            label="pass" if passed else "fail",
            score=1.0 if passed else 0.0,
            explanation=trajectory.serialize(),
        )
        if not passed:
            raise AssertionError("agent did not produce a final answer")
        return passed


class LimitsStepCount(EfficiencyAssertion):
    def __init__(self, logger: TraceAnnotator, max_steps: int) -> None:
        super().__init__(logger)
        self.max_steps = max_steps

    def check(self, trajectory: AgentTrajectory) -> bool:
        passed = len(trajectory.steps) <= self.max_steps
        self.logger.annotate(
            annotation_name="step_count",
            annotator_kind="CODE",
            label="pass" if passed else "fail",
            score=1.0 if passed else 0.0,
            metadata={"max_steps": self.max_steps, "actual_steps": len(trajectory.steps)},
        )
        return passed


scenario = EvalScenario(
    name="contract lookup",
    query="Find the renewal terms",
    category="retrieval",
    tier="smoke",
    metadata={"dataset": "contracts"},
)

logger = ...
runner = AgentEvalRunner(graph=agent.graph, logger=logger)
scorer = TrajectoryScorer().success(RequiresAnswer(logger)).expect(LimitsStepCount(logger, max_steps=3))
trajectory = await runner.run(scenario, scorer)

For LLM-as-judge checks, use LLMJudgeAssertion from agents_builder.eval.assertion. Its prompt must contain exactly {query} and {trajectory} placeholders, and each criterion is emitted as an LLM trace annotation.

Phoenix-backed annotation is available through PhoenixTraceAnnotator with PhoenixTraceAnnotatorSettings:

from agents_builder.eval.annotator import PhoenixTraceAnnotator
from agents_builder.settings import PhoenixTraceAnnotatorSettings

logger = PhoenixTraceAnnotator(
    PhoenixTraceAnnotatorSettings(
        module_path="agents_builder.eval.annotator.PhoenixTraceAnnotator",
        base_url="http://localhost:6006",
        project_name="agents-builder",
    )
)

Configuration

Configuration is built around Pydantic settings models in src/agents_builder/settings.py.

Important settings types:

  • AgentSettings
  • LLMFactorySettings
  • OllamaLLMSettings
  • OpenRouterLLMSettings
  • MCPServerSettings
  • TraceAnnotatorSettings
  • PhoenixTraceAnnotatorSettings
  • TraceAnnotatorSettings
  • PhoenixTraceAnnotatorSettings

The package-level default YAML path is:

/config/config.yaml

You can also construct objects directly from Python with .from_config(...) or load from YAML with .from_yaml(...).

Development Workflow

Quality checks already configured in the project:

  • ruff for linting and formatting
  • ty for type checking
  • pytest for unit, integration, and e2e tests
  • bandit for security checks

Recommended local loop:

make lint
make type-check
make test

Before merging broader changes:

make ci

Design Principles

This repository works best when changes stay aligned with a few constraints:

  • Prefer explicit graph nodes over magic orchestration layers.
  • Keep configuration typed and validated.
  • Use small runtime classes with narrow responsibilities.
  • Let tests describe behavior at the node and graph level.

If you are contributing code or using AI coding agents in this repository, read AGENTS.md next.

Metadata

Release files for agents-builder 4.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agents-builder 4.0.0
File Size Uploaded Provenance Actions
agents_builder-4.0.0.tar.gz 18.3 kB Without provenance Details for agents_builder-4.0.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agents-builder 4.0.0
File Interpreter ABI Platform Actions
agents_builder-4.0.0-py3-none-any.whl Python 3 none any Details for agents_builder-4.0.0-py3-none-any.whl

Total release size: 41.9 kB

Release files / agents_builder-4.0.0.tar.gz

Download URL agents_builder-4.0.0.tar.gz
Size 18.3 kB
Tags Source
SHA-256 checksum
How to use checksums
9f9fb3b7a246d52605af0a482381237ff00ba7534b62385da577f95f3e8015f8
BLAKE2b-256 checksum
How to use checksums
6ac962b00655a9af98001f638616bbc8f0f7c8288794b4fd658e11279ed892e3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.16 {"installer":{"name":"uv","version":"0.11.16","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / agents_builder-4.0.0-py3-none-any.whl

Download URL agents_builder-4.0.0-py3-none-any.whl
Size 23.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
fb0c784f53c5757f5c6cb0d89c8aaff8f84c78cfd21929d5d0b58655f1a9df34
BLAKE2b-256 checksum
How to use checksums
c664f9d2db1e6a090f6c18b6812f7b9c3b1fa0568c0422b431f9ae1a41f04ba0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.11.16 {"installer":{"name":"uv","version":"0.11.16","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for agents-builder 4.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page