agently-devtools 0.1.5

Observation and developer tooling companion package for Agently

agently-devtools

agently-devtools is the local observation, evaluation, and playground companion package for agently.

Install

pip install -U agently agently-devtools

Compatible with agently >=4.1.0.2,<4.2.0.

Compatibility resolution uses the current Agently offline manifest first, then the Agently release registry, then a bounded legacy fallback for 4.1.0.2 through 4.1.1.

Python >=3.10

Start

agently-devtools start

Default local address:

• Console: http://127.0.0.1:15596/
• Ingest: http://127.0.0.1:15596/observation/ingest

Explicit listener options:

agently-devtools start --host 0.0.0.0 --port 15596 --path /observation/ingest --no-open

Observation Bridge

from agently import Agently
from agently_devtools import ObservationBridge

bridge = ObservationBridge(
    app_id="your_app_id",
    group_id="your_group_id",
)
bridge.register(Agently)

Explicit endpoint styles are also supported:

ObservationBridge(host="127.0.0.1", port=15596, path="/observation/ingest")
ObservationBridge("https://devtools.example.com/observation/ingest")

Use selective watch when you do not want to upload every observation event:

bridge = ObservationBridge(
    app_id="your_app_id",
    auto_watch=False,
)
bridge.watch(agent, flow, lookup_reference)
bridge.register(Agently)

The bridge uses a background queue and coalesces high-frequency streaming events before upload, so passive observation does not block normal request output. For short-lived scripts, call await bridge.flush() before exit when you need all buffered events delivered.

TriggerFlow executions automatically publish their flow definition, so the execution graph can show the full static structure, including branches that have not been hit yet.

Scenario Evaluations

from agently_devtools import EvaluationBridge, EvaluationCase, EvaluationRunner

bridge = EvaluationBridge(
    base_url="http://127.0.0.1:15596",
    app_id="your_app_id",
    group_id="your_eval_group",
)
runner = EvaluationRunner(bridge=bridge)

binding = bridge.bind_agent_factory(
    lambda: build_agent(),
    suite_id="support-suite",
    target_name="support-agent",
)

report = runner.run(
    binding,
    cases=[
        EvaluationCase(case_id="refund", input="Need a refund for a duplicate payment."),
        EvaluationCase(case_id="shipping", input="Where is my shipment now?"),
    ],
    rules=[
        lambda record: bool(record.output),
        lambda record: record.error is None,
    ],
    rounds=2,
)

Use the binding helper that matches your target:

• bind_agent(...) / bind_agent_factory(...)
• bind_request(...) / bind_request_factory(...)
• bind_triggerflow(...) / bind_triggerflow_factory(...)

EvaluationBridge already records rounds into DevTools and keeps observed run linkage automatically. You do not need an extra ObservationBridge just for Scenario Evaluations.