AgentOps Accelerator

Evaluate. Ship. Observe. Operate.
Continuous evaluation, safety testing, observability, and release readiness for Microsoft Foundry agents.

Documentation | PyPI | VS Code Extension | Latest release

PyPI VS Code Extension CI Release License: MIT

AgentOps Accelerator helps Microsoft Foundry agent teams evaluate quality, prepare releases, monitor behavior, and operate reliably after launch. It gives you a practical starting point for agent operations, with Foundry integration as the default path and deeper setup guidance in the full docs.

Get started

python -m pip install agentops-accelerator
agentops init

agentops init starts a guided setup that creates your agentops.yaml and .agentops/ workspace.

Next, follow the tutorial that matches your agent type:

What it helps you do

Use AgentOps Accelerator when you need to:

  • Evaluate an agent before release
  • Compare changes across versions
  • Capture release evidence
  • Monitor agent quality and regressions
  • Give teams a repeatable way to operate agents responsibly in production

The accelerator keeps the local workflow simple, then points you to the full docs when you are ready to configure pipelines, dashboards, and release practices.

Learn more

For setup guides, tutorials, architecture, CI/CD guidance, Doctor checks, and evaluator reference, start with the documentation site:

https://aka.ms/agentops-accelerator

Run a first evaluation

az login
$env:AZURE_AI_FOUNDRY_PROJECT_ENDPOINT = "https://<resource>.services.ai.azure.com/api/projects/<project>"
$env:AZURE_OPENAI_ENDPOINT = "https://<openai-resource>.openai.azure.com"
$env:AZURE_OPENAI_DEPLOYMENT = "gpt-4o-mini"
agentops eval analyze
agentops eval run
agentops doctor --evidence-pack

For Foundry targets, use either project_endpoint: in agentops.yaml or AZURE_AI_FOUNDRY_PROJECT_ENDPOINT. Config wins when both are set.

Outputs land in .agentops/results/latest/:

  • results.json - machine-readable (versioned, stable schema)
  • report.md - human-readable, PR-friendly

Release evidence lands in .agentops/release/latest/:

  • evidence.json - machine-readable production-readiness projection
  • evidence.md - PR/release summary

Capture the first successful run as a baseline:

New-Item -ItemType Directory -Force .agentops\baseline | Out-Null
Copy-Item .agentops\results\latest\results.json .agentops\baseline\results.json

To see a visible comparison, publish a new agent version with a prompt that paraphrases instead of copying exact-answer requests, update agentops.yaml to that new name:version, and compare against the baseline:

agentops eval run --baseline .agentops/baseline/results.json

The report grows a Comparison vs Baseline section with per-metric deltas.

Commands

Install optional extras as needed: [agent] for Doctor/Cockpit and [mcp] for MCP.

  • agentops --version - show installed version.
  • agentops init - bootstrap config and seed data.
  • agentops eval analyze - check eval readiness.
  • agentops eval init - bootstrap an azd eval.yaml recipe and wire execution: azd.
  • agentops eval run [--baseline PATH] - run an evaluation.
  • agentops eval promote-traces --source FILE [--apply] - promote local trace export files.
  • agentops telemetry validate NAME - validate an Azure Monitor or Application Insights import.
  • agentops telemetry preview NAME --rows N - preview telemetry import rows.
  • agentops telemetry import NAME --apply - write the imported telemetry dataset.
  • agentops report generate - regenerate report.md.
  • agentops workflow analyze - recommend CI/CD shape.
  • agentops workflow generate - generate CI/CD workflows.
  • agentops skills install - install Copilot or Claude skills.
  • agentops mcp serve - start the MCP server.
  • agentops doctor [--evidence-pack] - run readiness checks.
  • agentops cockpit - open the local Cockpit.
  • agentops agent serve - serve Doctor as a Copilot Extension.

AgentOps Cockpit

agentops cockpit opens a localhost command center for the current workspace. It combines eval history, Doctor findings, workflow status, and links to the matching Foundry and Azure Monitor views.

Cockpit sections, in display order:

  • Foundry connection - project, tenant, agent, App Insights.
  • Foundry launchpad - links for the agent, project, and telemetry.
  • Observability readiness - tracing, evals, red team, alerts.
  • AgentOps Doctor - latest Doctor findings.
  • Eval gate summary - local and CI gate history.
  • Quality gate summary - score trends and regressions.
  • Production signal - App Insights health snapshot.
  • CI/CD Pipelines - GitHub Actions status.
  • Next actions - contextual recommendations.

Documentation

Contributing

See CONTRIBUTING.md for development, testing, and contribution guidance.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.13.0)

2 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for maxismus from gravatar.com maxismus Avatar for paulolacerda from gravatar.com paulolacerda

License

MIT License

Requires

Python >=3.11

Provides Extra

mcp agent

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

agentops_accelerator-0.8.6.tar.gz (11.0 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

agentops_accelerator-0.8.6-py3-none-any.whl (587.1 kB view details)

Uploaded Python 3

File details

Details for the file agentops_accelerator-0.8.6.tar.gz.

File metadata

  • Download URL: agentops_accelerator-0.8.6.tar.gz
  • Upload date:
  • Size: 11.0 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for agentops_accelerator-0.8.6.tar.gz
Algorithm Hash digest
SHA256 19fd9a8f9e20bed6df4cc50a2cb0999e2b9a7d1bbc890ea1e349a5de0509399a
MD5 a2c3fbc7d8c6cde28b42cbd50e9b6b28
BLAKE2b-256 08518878fe31e1677dd545c0220371485b86b8da134db13c9238b2674d6128e1

See more details on using hashes here.

Provenance

The following attestation bundles were made for agentops_accelerator-0.8.6.tar.gz:

Publisher: release.yml on Azure/agentops

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file agentops_accelerator-0.8.6-py3-none-any.whl.

File metadata

File hashes

Hashes for agentops_accelerator-0.8.6-py3-none-any.whl
Algorithm Hash digest
SHA256 6a8c2a32abc6436503f3879d6d3755a8446793855ab26c26cbc0040ea9f77e5d
MD5 871dcb1a7650e98f7d26c3cf0d45a905
BLAKE2b-256 e1c471012c47674808eff74b243b941e276256c486babc3201b38b3ebf8bb750

See more details on using hashes here.

Provenance

The following attestation bundles were made for agentops_accelerator-0.8.6-py3-none-any.whl:

Publisher: release.yml on Azure/agentops

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.13.0

2 files

0.12.0

2 files

0.11.0

2 files

0.10.0

2 files

0.9.0

2 files

0.8.8

2 files

0.8.7

2 files

This release

0.8.6 This release

2 files

0.8.5

2 files

0.8.4

2 files

0.8.3

2 files

0.8.2

2 files

0.8.1

2 files

0.8.0

2 files

0.7.1

2 files

0.7.0

2 files

0.6.0

2 files

0.5.3

2 files

0.5.2

2 files

0.5.1

2 files

0.5.0

2 files

0.4.5

2 files

0.4.4

2 files

0.4.3

2 files

0.4.2

2 files

0.4.1

2 files

0.3.23

2 files

0.3.22

2 files

0.3.21

2 files

0.3.20

2 files

0.3.19

2 files

0.3.18

2 files

0.3.16

2 files

0.3.15

2 files

0.3.14

2 files

0.3.13

2 files

0.3.12

2 files

0.3.11

2 files

0.3.10

2 files

0.3.9

2 files

0.3.8

2 files

0.3.7

2 files

0.3.6

2 files

0.3.5

2 files

0.3.4

2 files

0.3.3

2 files

0.3.2

2 files

0.3.1

2 files

0.3.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page