agently-devtools 0.1.3

Observation and developer tooling companion package for Agently

agently-devtools

agently-devtools is the local observation, evaluation, and playground companion package for agently.

Install

pip install -U agently agently-devtools

Compatible with agently >=4.1.0.2,<4.2.0.

Python >=3.10

Start

agently-devtools start

Default local address:

• Console: http://127.0.0.1:15596/
• Ingest: http://127.0.0.1:15596/observation/ingest

Explicit listener options:

agently-devtools start --host 0.0.0.0 --port 15596 --path /observation/ingest --no-open

Observation Bridge

from agently import Agently
from agently_devtools import ObservationBridge

bridge = ObservationBridge(
    app_id="your_app_id",
    group_id="your_group_id",
)
bridge.register(Agently)

Explicit endpoint styles are also supported:

ObservationBridge(host="127.0.0.1", port=15596, path="/observation/ingest")
ObservationBridge("https://devtools.example.com/observation/ingest")

Use selective watch when you do not want to upload every runtime event:

bridge = ObservationBridge(
    app_id="your_app_id",
    auto_watch=False,
)
bridge.watch(agent, flow, lookup_reference)
bridge.register(Agently)

TriggerFlow executions automatically publish their flow definition, so the execution graph can show the full static structure, including branches that have not been hit yet.

Scenario Evaluations

from agently_devtools import EvaluationBridge, EvaluationCase, EvaluationRunner

bridge = EvaluationBridge(
    base_url="http://127.0.0.1:15596",
    app_id="your_app_id",
    group_id="your_eval_group",
)
runner = EvaluationRunner(bridge=bridge)

binding = bridge.bind_agent_factory(
    lambda: build_agent(),
    suite_id="support-suite",
    target_name="support-agent",
)

report = runner.run(
    binding,
    cases=[
        EvaluationCase(case_id="refund", input="Need a refund for a duplicate payment."),
        EvaluationCase(case_id="shipping", input="Where is my shipment now?"),
    ],
    rules=[
        lambda record: bool(record.output),
        lambda record: record.error is None,
    ],
    rounds=2,
)

Use the binding helper that matches your target:

• bind_agent(...) / bind_agent_factory(...)
• bind_request(...) / bind_request_factory(...)
• bind_triggerflow(...) / bind_triggerflow_factory(...)

EvaluationBridge already records rounds into DevTools and keeps observed run linkage automatically. You do not need an extra ObservationBridge just for Scenario Evaluations.