agently-devtools

agently-devtools is the local observation, evaluation, and playground companion package for agently.

Install

pip install -U agently agently-devtools

Compatible with agently >=4.1.0.2,<4.2.0.

Compatibility resolution uses the current Agently offline manifest first, then the Agently release registry, then a bounded legacy fallback for 4.1.0.2 through 4.1.1.

Python >=3.10

Start

agently-devtools start

Default local address:

Console: http://127.0.0.1:15596/
Ingest: http://127.0.0.1:15596/observation/ingest

Explicit listener options:

agently-devtools start --host 0.0.0.0 --port 15596 --path /observation/ingest --no-open

Observation Bridge

from agently import Agently
from agently_devtools import ObservationBridge

bridge = ObservationBridge(
    Agently,
    app_id="your_app_id",
    group_id="your_group_id",
)
bridge.watch(Agently)

Explicit endpoint styles are also supported:

ObservationBridge(Agently, host="127.0.0.1", port=15596, path="/observation/ingest")
ObservationBridge(Agently, endpoint="https://devtools.example.com/observation/ingest")

Use selective watch when you do not want to upload every observation event:

bridge = ObservationBridge(
    Agently,
    app_id="your_app_id",
)
bridge.watch(agent, flow, lookup_reference)

The older bridge = ObservationBridge(...); bridge.register(Agently) style remains compatible and emits a deprecation warning. Prefer binding the Agently object or event center in the constructor, then selecting global or scoped observation with bridge.watch(...).

The bridge uses a background queue and coalesces high-frequency streaming events before upload, so passive observation does not block normal request output. By default it uses EventCenter summary delivery with dispatch="background", high_frequency_only=True, emit_interval=0.25, max_items=50, and max_summary_items=12; uploads are batched with upload_batch_size=200 and upload_flush_interval=0.25. You can override those values with delivery_policy, upload_batch_size, and upload_flush_interval on ObservationBridge(...). For short-lived scripts, call await bridge.flush() before exit when you need all buffered events delivered. Use bridge.get_stats() after flush when an experiment report needs upload batch, coalescing, drop, retry, and pending-buffer diagnostics.

Use group_id to isolate scenario, experiment, or environment runs under the same app_id. Runtime Observation, recent events, run events, SSE, and WebSocket subscriptions all accept the same group scope, so unrelated experiments do not trigger each other's refreshes.

Observation event storage keeps raw RuntimeEvents for audit. HTTP recent-events, run-events, SSE, and WebSocket hot paths return compact payload views by default so large prompts, model requests, and stream payloads do not overload the browser. Compacted events include a devtools_view.raw_event_ref, and the web console uses that cold reference to open full raw payloads in a modal only when requested. Add payload_mode=raw to observation event query endpoints only for scoped raw debug output.

The listener uses a bounded, run-partitioned receiver pipeline so events from one run remain ordered across concurrent ingest requests and awaited async sinks. Agently-Stage privately owns dispatcher/worker lifetime; DevTools owns partitioning, pressure, acknowledgement, errors, storage, and subscriptions. Ingest success is returned only after processing completes. Tune with observation.ingest.worker_count=4, observation.ingest.queue_limit=64, and observation.ingest.shutdown_timeout_seconds=5.0. Shutdown drains admitted batches and warns before cooperatively cancelling workers on timeout.

Local log persistence is buffered so SQLite fsync does not run for every high-frequency observation event. By default DevTools flushes the log buffer every logs.storage.flush_interval_seconds=600, when logs.storage.flush_event_limit=5000 pending events are reached, on shutdown, and at rate-limited business-critical events such as completed, failed, blocked, or error observations. Tune with logs.storage.flush_on_critical_events, logs.storage.critical_flush_min_interval_seconds, and logs.storage.flush_on_exit. Runtime Observation queries continue to include pending buffered events before they are persisted.

TriggerFlow executions automatically publish their flow definition, so the execution graph can show the full static structure, including branches that have not been hit yet.

AgentTask action observations are consumed from agent_task.action.started, agent_task.action.completed, and agent_task.action.failed RuntimeEvent records. The web console groups them under the AgentTask iteration actions section when iteration metadata is present. Treat these records as display/log facts only; they are not route decisions, verifier results, quality scores, semantic relevance judgments, or task-completion acceptance. Runtime lifecycle status and business Task Outcome are displayed separately; a completed run lifecycle is not treated as accepted business output unless the AgentTask result evidence says so.

Model request usage is consumed from payload.model_request_telemetry.usage_summary when present, with compatible fallbacks to provider usage metadata. The web console displays usage for each model request and aggregates usage upward for the selected run branch and descendants. When provider token counts are unavailable, token fields display NaN; estimated input/output character lengths are shown as diagnostics, not as budget or quality gates.

Provider reasoning is reconstructed from model.reasoning.delta and finalized by model.reasoning.completed in a separate Reasoning tab. Explicit provider reasoning_tokens usage is displayed and aggregated independently; missing reasoning content or usage stays unavailable, and reasoning tokens are never estimated from text or added again to provider output/completion/total values.

Scenario Evaluations

from agently_devtools import EvaluationBridge, EvaluationCase, EvaluationRunner

bridge = EvaluationBridge(
    base_url="http://127.0.0.1:15596",
    app_id="your_app_id",
    group_id="your_eval_group",
)
runner = EvaluationRunner(bridge=bridge)

binding = bridge.bind_agent_factory(
    lambda: build_agent(),
    suite_id="support-suite",
    target_name="support-agent",
)

report = runner.run(
    binding,
    cases=[
        EvaluationCase(case_id="refund", input="Need a refund for a duplicate payment."),
        EvaluationCase(case_id="shipping", input="Where is my shipment now?"),
    ],
    rules=[
        lambda record: bool(record.output),
        lambda record: record.error is None,
    ],
    rounds=2,
)

Use the binding helper that matches your target:

bind_agent(...) / bind_agent_factory(...)
bind_request(...) / bind_request_factory(...)
bind_triggerflow(...) / bind_triggerflow_factory(...)

EvaluationBridge already records rounds into DevTools and keeps observed run linkage automatically. You do not need an extra ObservationBridge just for Scenario Evaluations.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

agently_devtools-0.1.11-cp313-cp313-win_amd64.whl (1.6 MB view details)

Uploaded Aug 17, 2026 CPython 3.13Windows x86-64

agently_devtools-0.1.11-cp313-cp313-win32.whl (1.5 MB view details)

Uploaded Aug 17, 2026 CPython 3.13Windows x86

agently_devtools-0.1.11-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (5.0 MB view details)

Uploaded Aug 17, 2026 CPython 3.13manylinux: glibc 2.17+ x86-64manylinux: glibc 2.28+ x86-64

agently_devtools-0.1.11-cp313-cp313-macosx_11_0_arm64.whl (1.7 MB view details)

Uploaded Aug 17, 2026 CPython 3.13macOS 11.0+ ARM64

agently_devtools-0.1.11-cp313-cp313-macosx_10_13_x86_64.whl (1.7 MB view details)

Uploaded Aug 17, 2026 CPython 3.13macOS 10.13+ x86-64

agently_devtools-0.1.11-cp312-cp312-win_amd64.whl (1.6 MB view details)

Uploaded Aug 17, 2026 CPython 3.12Windows x86-64

agently_devtools-0.1.11-cp312-cp312-win32.whl (1.5 MB view details)

Uploaded Aug 17, 2026 CPython 3.12Windows x86

agently_devtools-0.1.11-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (5.0 MB view details)

Uploaded Aug 17, 2026 CPython 3.12manylinux: glibc 2.17+ x86-64manylinux: glibc 2.28+ x86-64

agently_devtools-0.1.11-cp312-cp312-macosx_11_0_arm64.whl (1.7 MB view details)

Uploaded Aug 17, 2026 CPython 3.12macOS 11.0+ ARM64

agently_devtools-0.1.11-cp312-cp312-macosx_10_13_x86_64.whl (1.7 MB view details)

Uploaded Aug 17, 2026 CPython 3.12macOS 10.13+ x86-64

agently_devtools-0.1.11-cp311-cp311-win_amd64.whl (1.6 MB view details)

Uploaded Aug 17, 2026 CPython 3.11Windows x86-64

agently_devtools-0.1.11-cp311-cp311-win32.whl (1.6 MB view details)

Uploaded Aug 17, 2026 CPython 3.11Windows x86

agently_devtools-0.1.11-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (5.0 MB view details)

Uploaded Aug 17, 2026 CPython 3.11manylinux: glibc 2.17+ x86-64manylinux: glibc 2.28+ x86-64

agently_devtools-0.1.11-cp311-cp311-macosx_11_0_arm64.whl (1.7 MB view details)

Uploaded Aug 17, 2026 CPython 3.11macOS 11.0+ ARM64

agently_devtools-0.1.11-cp311-cp311-macosx_10_9_x86_64.whl (1.7 MB view details)

Uploaded Aug 17, 2026 CPython 3.11macOS 10.9+ x86-64

agently_devtools-0.1.11-cp310-cp310-win_amd64.whl (1.6 MB view details)

Uploaded Aug 17, 2026 CPython 3.10Windows x86-64

agently_devtools-0.1.11-cp310-cp310-win32.whl (1.6 MB view details)

Uploaded Aug 17, 2026 CPython 3.10Windows x86

agently_devtools-0.1.11-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (4.8 MB view details)

Uploaded Aug 17, 2026 CPython 3.10manylinux: glibc 2.17+ x86-64manylinux: glibc 2.28+ x86-64

agently_devtools-0.1.11-cp310-cp310-macosx_11_0_arm64.whl (1.7 MB view details)

Uploaded Aug 17, 2026 CPython 3.10macOS 11.0+ ARM64

agently_devtools-0.1.11-cp310-cp310-macosx_10_9_x86_64.whl (1.7 MB view details)

Uploaded Aug 17, 2026 CPython 3.10macOS 10.9+ x86-64

File details

Details for the file agently_devtools-0.1.11-cp313-cp313-win_amd64.whl.

File metadata

Download URL: agently_devtools-0.1.11-cp313-cp313-win_amd64.whl
Upload date: Aug 17, 2026
Size: 1.6 MB
Tags: CPython 3.13, Windows x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for agently_devtools-0.1.11-cp313-cp313-win_amd64.whl
Algorithm	Hash digest
SHA256	`569814780fcba3d8d9e311c19afcef4b0194a253b991210cc3aa9b1a426309f0`
MD5	`5b93c786e5c32712083817969993cf70`
BLAKE2b-256	`7710568ac26d907d2f3d18f8e2b07d3d6974f491eaeb532c0455a99112709735`

Algorithm	Hash digest
SHA256	`981858e5f2c97b63287e80270cc7ecaf100c74fa096e92b48703bb43450898cc`
MD5	`b4d26122985ab1dc06f0180db4296cf3`
BLAKE2b-256	`efb9c17f1e23dd3fb686c9ad7f30e75387b0e45f282770b62cce92a27ed17026`

Algorithm	Hash digest
SHA256	`2f14b3f4699c6d98acdcbea5ef47cadcffcaae6b1395b592b4bc2114273d2a41`
MD5	`8f3fee7f764c8ee34aeb37326efc8e08`
BLAKE2b-256	`57e7a64c6daa6c3ea5b32f912e67bc246e2505fb1ac91cd0671b0148aee8af24`

Algorithm	Hash digest
SHA256	`6cc4a47529e0d78f6f73df8a7af691777b3392c61da576cdffe78b4de2a88ee0`
MD5	`a8e8bffd91abd7e375158dd52ade2948`
BLAKE2b-256	`04c683c132b0f2808ae4603fd9a212d15aca3d5c046a409108346e0ce803d032`

Algorithm	Hash digest
SHA256	`ae73fd82c68ed511029dc5fa86cb1ad22e1cadf3aec2825b547bd9c204f1f47e`
MD5	`05165f5166fb8901643128a2eca0a409`
BLAKE2b-256	`8f5e5e9d39680bba656db8cfda94c85dc748815402f8c8665720167d6025d5f8`

Algorithm	Hash digest
SHA256	`ba270bc0535d158dbf07e17bdf9c015af9458c333ef691a92815412dd409b8d3`
MD5	`659a19e3c330fff2ebfa9b51fd13d98c`
BLAKE2b-256	`cf8c4d7b4665265e55174f2402f70da46fbc8bed21ab4c444c116a89abe1a422`

Algorithm	Hash digest
SHA256	`190624bd609f6a73ca66a8fc6b87d54f97c32557b142442413bd68d36f36ceaf`
MD5	`256e01007db3b00c565e5f13d6f8da63`
BLAKE2b-256	`3df2540922c8697bfe39a01bc9110aced1b6826a9763723d5a4985970c646457`

Algorithm	Hash digest
SHA256	`4145b0c7db5501499f9173f9e603d54e41180927fda959aaf7bf75ba09ba9adb`
MD5	`c64e487f15043fbcb6c22f96da9d1e5e`
BLAKE2b-256	`92e5ee0152afa9dcf7c0fd6b75f2aa49dca4b95daacf889fd22fb5a0fc24f47b`

Algorithm	Hash digest
SHA256	`60f5950d608c3b1593f08b57a562989f44d8b7e9a28bf0447d715b0d7036737a`
MD5	`4508ebb285c77e736632351b9d1af977`
BLAKE2b-256	`ab9d22f4452af17fad49cba34e288e0428c5a2aee34671bdcb93a7e6d2271222`

Algorithm	Hash digest
SHA256	`62f0a0282d7e3fd2e2635c5c7d53027ef9a33bbe77c34d3acb895b59accac7dc`
MD5	`12d641cadc43e6f7ca547ff37e6f5c16`
BLAKE2b-256	`8002431821fbd991ade6bb77fae642566f75ce144247fd69bbcf4907846dcb56`

Algorithm	Hash digest
SHA256	`62dc8a4a3e196fb1c3720667d22ba1e5517fc9864e49c85751bb66eb16458466`
MD5	`f5305fc4c6a07c8863807da6b7611bd2`
BLAKE2b-256	`01f65327aa8248d2231240be853d567e791dfc8afc00d6638d181c9a2b8d18ea`

Algorithm	Hash digest
SHA256	`fa0275e8d5cc0d30744d638c1e2b059d25d1bdc2e5673bdedfa169de36e842e7`
MD5	`5f956062bd4f3b732b64338a87cf32a2`
BLAKE2b-256	`30d0c72914c01279ede1c9622e2074fee4b49439979d8e9def8e4f9d8a0f2110`

Algorithm	Hash digest
SHA256	`719580347ac377014d6d312e8099f499dae5b707568ba66745e9014c53a16cca`
MD5	`3b1cb341e8f65f70ed59b514184f9348`
BLAKE2b-256	`0f83a5597396fbf2350f7b974cbf973b845f6948fa4c10bf041bf5ef7cf75e55`

Algorithm	Hash digest
SHA256	`c7ee3ddaed9f11d82e066b32f1c5b84351c49137170ad0276e05f04f1124d33e`
MD5	`e016ee375583e8fea8c09205ba4f4134`
BLAKE2b-256	`df19ddf8dac3857f0dfbf96abf393d86a4788065155d50dc31e9b1ba24c38255`

Algorithm	Hash digest
SHA256	`3daa159cf31623a2d934ed8eb6d04b18ca34d83636651df80fb23d6b99cfb1c3`
MD5	`33e1cecf0c76087d2a48233e8c1ab0d6`
BLAKE2b-256	`989657eac3ff1856c90d08dd2db56d72d1c17c1070c48423a422c1ab037269dd`

Algorithm	Hash digest
SHA256	`b39783c283f8669667f4df08d44eb012a18c79847ddb86170bf5b8b54581d5bf`
MD5	`28d5c420edf258d8111356a9bfbc40b1`
BLAKE2b-256	`3463cbe617b491b48311316a39fea89e9080a3ed7997648ceee0953881bbeaf3`

Algorithm	Hash digest
SHA256	`3e707fb68b851e48cad1284e38e8dee9f21f0e70914ba284e5edca34ce91ea62`
MD5	`f599f58090378d084fc6b81734a9003f`
BLAKE2b-256	`0be763543e5edbd323d01cba33762b1ddb968dba45c91b0748384689d90e9c23`

Algorithm	Hash digest
SHA256	`d48981febd483d4799e329588e703fa220a009ee94351ee8baa656b1cfd415de`
MD5	`de0a542a009cea63754d143879849e8a`
BLAKE2b-256	`c6f6e4ca2a5dc81beda0a8b95f2441b5753e302f7ce592461a2b40c99357b06e`

Algorithm	Hash digest
SHA256	`83cc11feff20d0c7aff107c2c154073081365ee4cf09b160d9feeef6d681758c`
MD5	`3f8d487b4aba91d672b33094e83b2125`
BLAKE2b-256	`4da8a15ca1c49ad929f2164117ce026dc0aab56ddd8f2cab7c49519cf78b1ced`

Algorithm	Hash digest
SHA256	`4154c257135656bebc7aa95e307dbbfe9a320ed523c832d4ee3d3218d615eb3e`
MD5	`cce3d4d6f7c5510368a997d98d385590`
BLAKE2b-256	`80c1e7e7e474ac3447b18f5d9e94b3c83de68a725a40588b21a6a098701f44c3`

agently-devtools 0.1.11

agently-devtools

Install

Start

Observation Bridge

Scenario Evaluations

Key dates

1 maintainer

Credits

Requires

Provides Extra

Download files

Source Distributions

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata