AgentOps Accelerator
Evaluate. Ship. Operate.
Continuous evaluation, safety testing, observability, and release readiness for Microsoft Foundry agents.
Documentation | PyPI | VS Code Extension | Latest release
AgentOps Accelerator helps Microsoft Foundry agent teams evaluate quality, prepare releases, monitor behavior, and operate reliably after launch. It gives you a practical starting point for agent operations, with Foundry integration as the default path and deeper setup guidance in the full docs.
Get started
python -m pip install "agentops-accelerator[cockpit]"
agentops init
Use the [cockpit] option when you want the local Cockpit.
To install a specific release directly from GitHub:
python -m pip install "agentops-accelerator[cockpit] @ git+https://github.com/Azure/agentops.git@vX.Y.Z"
Replace vX.Y.Z with the desired release tag.
agentops init starts a guided setup that creates your agentops.yaml and
.agentops/ workspace.
Next, follow the tutorial that matches your agent type:
- Open the documentation entry point: https://aka.ms/agentops-accelerator
- Choose the Prompt Agent or Hosted/HTTP Agent tutorial from Tutorials.
What it helps you do
Use AgentOps Accelerator when you need to:
- Evaluate an agent before release
- Compare changes across versions
- Capture release evidence
- Monitor agent quality and regressions
- Give teams a repeatable way to operate agents responsibly in production
The accelerator keeps the local workflow simple, then points you to the full docs when you are ready to configure pipelines, dashboards, and release practices.
Learn more
For setup guides, tutorials, architecture, CI/CD guidance, Doctor checks, and evaluator reference, start with the documentation site:
https://aka.ms/agentops-accelerator
Run a first evaluation
az login
$env:AZURE_AI_FOUNDRY_PROJECT_ENDPOINT = "https://<resource>.services.ai.azure.com/api/projects/<project>"
$env:AZURE_OPENAI_ENDPOINT = "https://<openai-resource>.openai.azure.com"
$env:AZURE_OPENAI_DEPLOYMENT = "gpt-4o-mini"
agentops eval analyze
agentops eval run
agentops doctor --evidence-pack
For Foundry targets, use either project_endpoint: in agentops.yaml or
AZURE_AI_FOUNDRY_PROJECT_ENDPOINT. Config wins when both are set.
Outputs land in .agentops/results/latest/:
results.json- machine-readable (versioned, stable schema)report.md- human-readable, PR-friendly
Release evidence lands in .agentops/release/latest/:
evidence.json- machine-readable production-readiness projectionevidence.md- PR/release summary
Capture the first successful run as a baseline:
New-Item -ItemType Directory -Force .agentops\baseline | Out-Null
Copy-Item .agentops\results\latest\results.json .agentops\baseline\results.json
To see a visible comparison, publish a new agent version with a prompt
that paraphrases instead of copying exact-answer requests, update
agentops.yaml to that new name:version, and compare against the
baseline:
agentops eval run --baseline .agentops/baseline/results.json
The report grows a Comparison vs Baseline section with per-metric deltas.
Commands
Install optional extras as needed: [cockpit] for Doctor/Cockpit and [mcp] for MCP.
agentops --version- show installed version.agentops init- bootstrap config and seed data.agentops eval analyze- check eval readiness.agentops eval init- bootstrap an azdeval.yamlrecipe and wireexecution: azd.agentops eval run [--baseline PATH]- run an evaluation.agentops eval promote-traces --source FILE [--apply]- promote local trace export files.agentops telemetry validate NAME- validate an Azure Monitor or Application Insights import.agentops telemetry preview NAME --rows N- preview telemetry import rows.agentops telemetry import NAME --apply- write the imported telemetry dataset.agentops report generate- regeneratereport.md.agentops workflow analyze- recommend CI/CD shape.agentops workflow generate- generate CI/CD workflows.agentops skills install- install Copilot or Claude skills.agentops mcp serve- start the MCP server.agentops doctor [--evidence-pack]- run readiness checks.agentops cockpit- open the local Cockpit.
AgentOps Cockpit
agentops cockpit opens a localhost command center for the current workspace.
It combines eval history, Doctor findings, workflow status, and links to the
matching Foundry and Azure Monitor views.
Cockpit sections, in display order:
- Foundry connection - project, tenant, agent, App Insights.
- Foundry launchpad - links for the agent, project, and telemetry.
- Observability readiness - tracing, evals, red team, alerts.
- AgentOps Doctor - latest Doctor findings.
- Eval gate summary - local and CI gate history.
- Quality gate summary - score trends and regressions.
- Production signal - App Insights health snapshot.
- CI/CD Pipelines - GitHub Actions status.
- Next actions - contextual recommendations.
Documentation
- Foundry Prompt Agent tutorial - use this when the Foundry target is
agent: name:version. Walks the sandbox to dev journey with a PR gate. - Hosted Agent tutorial - use this when Foundry runs your agent code as a managed hosted runtime behind a stable endpoint.
- HTTP Agent tutorial - use this when the target is an HTTP service you operate behind your own URL.
- End-to-end tutorial - extends either of the above with the full sandbox to dev to qa to prod promotion, Foundry red-team scans, and trace-to-regression promotion.
- Evaluation paths - create datasets and choose local or Foundry cloud execution.
- Azure Storage datasets - optionally read private Blob or ADLS Gen2 datasets with Azure identity.
- Core concepts
- How it works
- Doctor explained
- CI/CD with GitHub Actions
- Built-in evaluator reference
- Release process
Contributing
See CONTRIBUTING.md for development, testing, and contribution guidance.
2 maintainers
Metadata
2 maintainers
Release files for agentops-accelerator 0.15.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| agentops_accelerator-0.15.1.tar.gz | 15.7 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| agentops_accelerator-0.15.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 16.3 MB
Release files / agentops_accelerator-0.15.1.tar.gz
| Download URL | agentops_accelerator-0.15.1.tar.gz |
|---|---|
| Size | 15.7 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d527ce19908d94594f198eda0f3c663feb5dfe2d7889f76cf5e31392dca47f03
|
|
BLAKE2b-256 checksum How to use checksums |
a619c1a9a9caa9468c30a787a31f9556704931786f527e6c965ea3c6edadf028
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 7, 2026.
Transparency logRelease files / agentops_accelerator-0.15.1-py3-none-any.whl
| Download URL | agentops_accelerator-0.15.1-py3-none-any.whl |
|---|---|
| Size | 597.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
61f6927eca9ce0bb1e211440be1f7427e83b4faa4b0c71a62275300e6024a9c7
|
|
BLAKE2b-256 checksum How to use checksums |
9274952372b874dc8e4117a9567a37ed8c0bb806f1bebc6289be61e0dc8a3ea8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 7, 2026.
Transparency log
