Applied Labs CLI
CLI and client library for Applied Labs AI support agents.
Installation
pip install applied-cli
CLI Usage
# Authenticate
applied login
# List agents
applied agents
applied agents --format json
# Find and inspect conversations
applied conversation-find --query "refund" --limit 5
applied conversations --resolution escalated --view detail --limit 10
applied conversation <id> --messages --format json
# Query tickets
applied tickets --status open
# Knowledge base
applied knowledge --type qa --search "refund"
applied knowledge <id> --format json
applied knowledge --source "https://help.example.com" --format csv
applied knowledge-diff --source "https://help.example.com"
applied content list --source "https://help.example.com"
applied content-update <content_id> --text "Corrected source text"
applied content-resync <content_id> --wait
applied knowledge-pin <id>
applied knowledge-link <response_id> --content <content_id>
applied knowledge-protect <id>
applied knowledge-unprotect <id>
# Taxonomy
applied taxonomy --type topics
applied taxonomy-counts --start 2026-03-01 --end 2026-03-18 --format csv
# Analytics
applied analytics-report --view overview_aggregate_metrics --model conversation --start 2026-04-01 --end 2026-04-30 --format json
applied analytics --group-by topic --metrics count --start 2026-04-01 --end 2026-04-30 --format json
applied analytics --group-by intent --metrics count --start 2026-04-01 --end 2026-04-30 --format json
applied metrics --metric-name conversation.resolve --start 2026-04-01 --end 2026-04-30 --period day --format json
analytics-report returns the selected report payload, not necessarily a { "rows": [...] }
object. analytics returns grouped rows and currently supports --metrics count.
Raw analytics SQL is not available through the public CLI surface.
Benchmarks & Scenarios
LEGACY surface. Benchmarks and scenarios are the older regression-suite system. The current evaluation system is the /evals surface (feedback + simulations + test data), driven by the
simulation_test_*,test_customer_*, andsimulation_run_*tools (MCP //v1/simulation-tests/,/v1/test-customers/,/v1/simulation-runs/). Thefeedbackfield onscenario_update/scenario_run_updateis a legacy free-text note, NOT the new Judgment+Comment feedback surface. Use the /evals tools for new evaluation work; the tools below are for maintaining existing benchmark suites.
A benchmark is a named regression suite; a scenario is one test conversation
(built from a real input_conversation_id) that can belong to one or more benchmarks.
The typical loop is: build a suite → run it → review the pass rate → fix → re-run.
# Inspect benchmarks and their scenarios
applied benchmarks --agent-id <agent_id> --format json
applied benchmark <benchmark_id> --format json
applied scenarios --benchmark-id <benchmark_id> --format json
# Build a suite
applied benchmark-create --agent-id <agent_id> --name "Cancel Regression"
applied scenario-create --input-conversation-id <conversation_id> --name "<name>" \
--benchmark-id <benchmark_id>
# Build a suite fast from several real conversations at once. Each scenario is
# named from its conversation's title (or "<prefix> N" with --name-prefix).
applied scenario-create-bulk --conversation-ids <id1>,<id2>,<id3> \
--benchmark-id <benchmark_id>
# Port a suite to another agent (e.g. email -> chat). Cross-agent recreates the
# scenarios under the destination agent; same-agent just tags them in.
# Dry-run by default; add --apply to write.
applied benchmark-clone <source_benchmark_id> --dest-benchmark-name "Chat Regression" \
--target-agent-id <chat_agent_id> --apply
# Run a benchmark and wait for results in one command.
# --contact-email runs as a contact that has an email, fixing
# "Email is not present in the conversation" on test conversations.
applied scenario-bulk-run --benchmark-id <benchmark_id> \
--contact-email test@example.com --wait
applied scenario-bulk-status <job_id> --include-runs --format json
# Kill a stuck bulk run (deletes its queued/running runs; finished runs preserved)
applied scenario-bulk-cancel <job_id> --apply
# Review pass/fail health (pass_status reflects the latest run per scenario)
applied benchmark-results <benchmark_id> --format json
# Portfolio go/no-go: pass rates across all of an agent's benchmarks at a glance
applied benchmarks --agent-id <agent_id> --with-results --format json
# Rate scenarios as you evaluate
applied scenario-update <scenario_id> --pass-status pass --feedback "<note>"
# Safe delete — refuses to wipe scenarios unless you opt in
applied benchmark-delete <benchmark_id> --detach-scenarios # preserve scenarios
applied benchmark-delete <benchmark_id> --force # cascade delete
# Recover deleted benchmark/scenario rows from a local PITR export
applied scenario-recover-catalog --recovery-dir <dir> --apply
Deleting a benchmark cascades and permanently deletes its scenarios and runs, so
benchmark-delete refuses a non-empty benchmark unless you pass --detach-scenarios
(unlink the scenarios first so they survive under their agent) or --force.
Library Usage
from applied_cli import AppliedClient, tools
# Create client
client = AppliedClient(token="al_xxx")
# Use tools
agents = await tools.agent_list(client, output_format="csv")
conversations = await tools.conversation_query(
client,
filters={"resolution": "escalated"},
view="search",
limit=20,
)
Tools
| Tool | Description |
|---|---|
agent_list |
List all AI agents |
conversation_find |
Find conversations with compact agent-friendly fields |
conversation_get |
Get single conversation with messages |
conversation_query |
Search/filter conversations |
ticket_query |
Search/filter tickets |
knowledge_list |
List knowledge base items |
taxonomy_list |
List topics, intents, and flags |
taxonomy_counts |
Aggregate conversation counts by topic and intent |
analytics_report |
Read standard dashboard/report analytics views |
analytics_query |
Aggregate supported conversation dimensions with count |
metrics_query |
Roll up named metric events |
benchmark_clone |
Copy all scenarios from one benchmark into another |
benchmark_delete |
Delete a benchmark (guards against wiping scenarios) |
benchmark_results |
Pass/fail/unrated tally and pass rate for a benchmark |
benchmark_list |
List benchmarks (with per-benchmark pass rates via with_results) |
scenario_create_bulk |
Build scenarios from several conversations at once |
scenario_bulk_run |
Run scenarios (contact override + wait-to-completion) |
scenario_bulk_cancel |
Cancel a stuck bulk run's queued/running scenario runs |
review_run |
Grade real conversations with the AI Reviewer (pass/fail) |
review_status |
Verdict pairs + the Reviewer's reason for a set of conversations |
review_set_verdict |
Record the human verdict with a teaching reason |
review_teach |
Teach the Reviewer a rule directly |
review_make_test |
Turn a flagged conversation into a proposed regression test |
review_proposals_list |
The suite change-inbox (proposed adds/edits/retires/merges) |
review_proposals_decide |
Accept or dismiss a proposed test |
The full review_* namespace also includes review_get, review_clear_verdict,
review_add_note, review_trust, and review_lessons — the outer review loop:
grade real traffic, confirm/overrule with a reason that teaches the Reviewer, and
promote failures into the simulation suite as tests.
Examples
# Find conversations using the default compact search view
await tools.conversation_find(
client,
query="refund",
output_format="csv"
)
# Find escalated conversations
await tools.conversation_query(
client,
filters={"resolution": "escalated"},
view="detail",
output_format="csv"
)
# Get conversation with transcript
await tools.conversation_get(
client,
conversation_id="abc-123",
include_messages=True,
message_limit=50,
output_format="json"
)
# Search knowledge base
await tools.knowledge_list(
client,
kb_type="qa",
search="refund policy",
output_format="json"
)
Development
# Install dev dependencies
pip install -e ".[dev]"
# Run tests
pytest
# Lint
ruff check --fix .
ruff format .
License
MIT
Metadata
Release files for applied-cli 0.7.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| applied_cli-0.7.1.tar.gz | 221.0 kB | With provenance | Details for applied_cli-0.7.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| applied_cli-0.7.1-py3-none-any.whl | Python 3 | none | any | Details for applied_cli-0.7.1-py3-none-any.whl |
Total release size: 398.4 kB
Release files / applied_cli-0.7.1.tar.gz
| Download URL | applied_cli-0.7.1.tar.gz |
|---|---|
| Size | 221.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8cf1636c03fef501e9fa9fa14b3083d4f26611b718006c27a1505f26508f3690
|
|
BLAKE2b-256 checksum How to use checksums |
c30f9d88952a8396d896030c5347220092c610a6d0be3d3e8f67e2bb2b87957a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 16, 2026.
Transparency logRelease files / applied_cli-0.7.1-py3-none-any.whl
| Download URL | applied_cli-0.7.1-py3-none-any.whl |
|---|---|
| Size | 177.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0ae7c65d78ee4c0cc5a30e6f389e6c73827e2c7eb60a78e9e15f0c331c2179f6
|
|
BLAKE2b-256 checksum How to use checksums |
4e06fdb742d7d7c9d64c959d7c84381bbf2d2a6eb3759a62f95d01e40ad2ee49
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 16, 2026.
Transparency logSecurity for applied-cli 0.7.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for applied_cli-0.7.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 16, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.