agently-devtools

agently-devtools is the local observation, evaluation, and playground companion package for agently.

Install

pip install -U agently agently-devtools

Compatible with agently >=4.1.0.2,<4.2.0.

Compatibility resolution uses the current Agently offline manifest first, then the Agently release registry, then a bounded legacy fallback for 4.1.0.2 through 4.1.1.

Python >=3.10

Start

agently-devtools start

Default local address:

• Console: http://127.0.0.1:15596/
• Ingest: http://127.0.0.1:15596/observation/ingest

Explicit listener options:

agently-devtools start --host 0.0.0.0 --port 15596 --path /observation/ingest --no-open

Observation Bridge

from agently import Agently
from agently_devtools import ObservationBridge

bridge = ObservationBridge(
    Agently,
    app_id="your_app_id",
    group_id="your_group_id",
)
bridge.watch(Agently)

Explicit endpoint styles are also supported:

ObservationBridge(Agently, host="127.0.0.1", port=15596, path="/observation/ingest")
ObservationBridge(Agently, endpoint="https://devtools.example.com/observation/ingest")

Use selective watch when you do not want to upload every observation event:

bridge = ObservationBridge(
    Agently,
    app_id="your_app_id",
)
bridge.watch(agent, flow, lookup_reference)

The older bridge = ObservationBridge(...); bridge.register(Agently) style remains compatible and emits a deprecation warning. Prefer binding the Agently object or event center in the constructor, then selecting global or scoped observation with bridge.watch(...).

The bridge uses a background queue and coalesces high-frequency streaming events before upload, so passive observation does not block normal request output. By default it uses EventCenter summary delivery with dispatch="background", high_frequency_only=True, emit_interval=0.25, max_items=50, and max_summary_items=12; uploads are batched with upload_batch_size=200 and upload_flush_interval=0.25. You can override those values with delivery_policy, upload_batch_size, and upload_flush_interval on ObservationBridge(...). For short-lived scripts, call await bridge.flush() before exit when you need all buffered events delivered. Use bridge.get_stats() after flush when an experiment report needs upload batch, coalescing, drop, retry, and pending-buffer diagnostics.

Use group_id to isolate scenario, experiment, or environment runs under the same app_id. Runtime Observation, recent events, run events, SSE, and WebSocket subscriptions all accept the same group scope, so unrelated experiments do not trigger each other's refreshes.

Observation event storage keeps raw RuntimeEvents for audit. HTTP recent-events, run-events, SSE, and WebSocket hot paths return compact payload views by default so large prompts, model requests, and stream payloads do not overload the browser. Compacted events include a devtools_view.raw_event_ref, and the web console uses that cold reference to open full raw payloads in a modal only when requested. Add payload_mode=raw to observation event query endpoints only for scoped raw debug output.

The listener uses a bounded, run-partitioned receiver pipeline so events from one run remain ordered across concurrent ingest requests and awaited async sinks. Agently-Stage privately owns dispatcher/worker lifetime; DevTools owns partitioning, pressure, acknowledgement, errors, storage, and subscriptions. Ingest success is returned only after processing completes. Tune with observation.ingest.worker_count=4, observation.ingest.queue_limit=64, and observation.ingest.shutdown_timeout_seconds=5.0. Shutdown drains admitted batches and warns before cooperatively cancelling workers on timeout.

Local log persistence is buffered so SQLite fsync does not run for every high-frequency observation event. By default DevTools flushes the log buffer every logs.storage.flush_interval_seconds=600, when logs.storage.flush_event_limit=5000 pending events are reached, on shutdown, and at rate-limited business-critical events such as completed, failed, blocked, or error observations. Tune with logs.storage.flush_on_critical_events, logs.storage.critical_flush_min_interval_seconds, and logs.storage.flush_on_exit. Runtime Observation queries continue to include pending buffered events before they are persisted.

TriggerFlow executions automatically publish their flow definition, so the execution graph can show the full static structure, including branches that have not been hit yet.

AgentTask action observations are consumed from agent_task.action.started, agent_task.action.completed, and agent_task.action.failed RuntimeEvent records. The web console groups them under the AgentTask iteration actions section when iteration metadata is present. Treat these records as display/log facts only; they are not route decisions, verifier results, quality scores, semantic relevance judgments, or task-completion acceptance. Runtime lifecycle status and business Task Outcome are displayed separately; a completed run lifecycle is not treated as accepted business output unless the AgentTask result evidence says so.

Model request usage is consumed from payload.model_request_telemetry.usage_summary when present, with compatible fallbacks to provider usage metadata. The web console displays usage for each model request and aggregates usage upward for the selected run branch and descendants. When provider token counts are unavailable, token fields display unavailable; estimated input/output character lengths are shown as diagnostics, not as budget or quality gates.

Provider reasoning is reconstructed from model.reasoning.delta and finalized by model.reasoning.completed in a separate Reasoning tab. Explicit provider reasoning_tokens usage is displayed and aggregated independently; missing reasoning content or usage stays unavailable, and reasoning tokens are never estimated from text or added again to provider output/completion/total values.

Scenario Evaluations

from agently_devtools import EvaluationBridge, EvaluationCase, EvaluationRunner

bridge = EvaluationBridge(
    base_url="http://127.0.0.1:15596",
    app_id="your_app_id",
    group_id="your_eval_group",
)
runner = EvaluationRunner(bridge=bridge)

binding = bridge.bind_agent_factory(
    lambda: build_agent(),
    suite_id="support-suite",
    target_name="support-agent",
)

report = runner.run(
    binding,
    cases=[
        EvaluationCase(case_id="refund", input="Need a refund for a duplicate payment."),
        EvaluationCase(case_id="shipping", input="Where is my shipment now?"),
    ],
    rules=[
        lambda record: bool(record.output),
        lambda record: record.error is None,
    ],
    rounds=2,
)

Use the binding helper that matches your target:

• bind_agent(...) / bind_agent_factory(...)
• bind_request(...) / bind_request_factory(...)
• bind_triggerflow(...) / bind_triggerflow_factory(...)

EvaluationBridge already records rounds into DevTools and keeps observed run linkage automatically. You do not need an extra ObservationBridge just for Scenario Evaluations.

Experimental run reviews

DevTools 0.2 provides evidence-linked reviews, explicit flow/Prompt/expectation changes, handoff previews and rerun feedback. See the repository docs/reviews.md. Reviews do not dispatch agents or authorize code execution.