Azure Pipeline Validator

azure-pipeline-validator is a batteries‑included Azure DevOps YAML inspector that runs the same validations you rely on in the service, but locally. It combines three feedback loops:

1. yamllint – fast structural linting using a tuned configuration for Azure Pipelines quirks.
2. JSON Schema – offline validation against Microsoft’s published schema (/distributedtask/yamlschema).
3. Preview REST API – invokes POST .../_apis/pipelines/{id}/preview with yamlOverride, returning the real finalYaml and any validationResults that Azure DevOps would produce.

The CLI understands both single files and whole repositories, wraps templates automatically (steps/jobs/stages), and mirrors the live API response schema (including continuation_token).

Table of contents

• Azure Pipeline Validator
  • Table of contents
  • Features
  • Installation & invocation
  • Required environment
  • Usage examples
  • CLI reference
  • Output format
  • CI integration
  • Development workflow
  • Publishing the package
  • Troubleshooting
  • License

Features

• Template auto-wrapping – detects steps/jobs/stages templates, wrapping them into runnable pipelines before previewing.
• Schema caching – fetches the official schema once per run and reuses it for every file, keeping validation snappy.
• Rich reporting – console output shows pass/fail per file with the first offending message per stage.
• Toggleable stages – disable lint/schema/preview individually for quick iteration.
• Failure ergonomics – --fail-fast stops on first failure; exit codes are 0 (success) or 1 (any failures/API issues).
• Selective scanning – --exclude / -x lets you skip generated folders or files with glob patterns.
• UV-native – built with uv, so you can run it via uv run, uvx, or install it as a global tool.

Installation & invocation

Getting started (recommended)

1. Install uv (pick the command for your OS):

   • macOS / Linux

     curl -LsSf https://astral.sh/uv/install.sh | sh
     

   • Windows (PowerShell)

     powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
     

2. Pick the validations you need – nothing runs until you opt in:

   • Lint-only (no Azure creds needed):

     uvx azure-pipeline-validator --lint .
     

   • Schema check (downloads the public schema anonymously):

     uvx azure-pipeline-validator --schema workflows/ci.yml
     

   • Preview (requires AZDO_* + AZDO_PAT because it calls your org):

     uvx azure-pipeline-validator --preview --schema workflows/
     

Short flags -l, -s, and -p map to --lint, --schema, and --preview if you prefer brevity, and you can combine them as needed.

Global install with uv (install once, use anywhere):

uv tool install git+https://github.com/andrewmaspero/azure-pipeline-validator.git
azure-pipeline-validator --help

Once published to PyPI, you can also use:

uv tool install azure-pipeline-validator
azure-pipeline-validator --help

Pip install will also work once published (pip install azure-pipeline-validator).

Required environment

Environment variables (or their CLI equivalents) are only required when --preview/-p is enabled. Lint (--lint/-l) and schema (--schema/-s) checks run entirely offline using the public schema.

Export the same variables you would in an Azure Pipelines job, or pass them via the --azdo-* options:

Tips:

• You can set variables inline without shell-specific syntax: uvx azure-pipeline-validator AZDO_ORG=https://dev.azure.com/contoso AZDO_PROJECT=demo ...
• Every Azure option also has a CLI flag, e.g. uvx azure-pipeline-validator --azdo-org https://dev.azure.com/contoso --azdo-pat token workflows/.
• Already signed in via az devops login? The cached PAT from the Azure CLI DevOps extension (or AZURE_DEVOPS_EXT_PAT) is picked up automatically.
• Defaults configured via az devops configure --defaults organization=... project=... are used whenever AZDO_ORG/AZDO_PROJECT are missing.
• Inside Azure Pipelines you can skip AZDO_PAT by enabling “Allow scripts to access the OAuth token” and mapping it to SYSTEM_ACCESSTOKEN.

Discovering available projects

Need to remember project names? Run:

uvx azure-pipeline-validator projects --top 20

The command reuses the same credential discovery (env vars, az devops login, defaults configured via az devops configure).

Usage examples

Validate the entire repo with every check (requires AZDO_* + PAT):

uv run azure-pipeline-validator --lint --schema --preview . --repo-root $(pwd)

Validate a single template file using only the schema:

uv run azure-pipeline-validator --schema common/templates/steps/build.yml

Lint a directory quickly:

uv run azure-pipeline-validator --lint workflows/

Skip generated templates while linting:

uv run azure-pipeline-validator --lint workflows/ --exclude "workflows/generated"

CLI reference

Usage: azure-pipeline-validator [OPTIONS] [PATH]

Run yamllint, schema validation, and Azure preview against YAML files.

Arguments:
  PATH  File or directory to validate. Directories are scanned recursively for *.yml and *.yaml files.  [default: .]

Options:
  --repo-root PATH                     Base path used when resolving template references (defaults to CWD).
  --azdo-org URL                       Organization URL (overrides AZDO_ORG).
  --azdo-project NAME                  Project name (overrides AZDO_PROJECT).
  --azdo-pipeline-id ID                Pipeline ID used for preview (overrides AZDO_PIPELINE_ID).
  --azdo-pat TOKEN                     PAT or OAuth token (overrides AZDO_PAT / SYSTEM_ACCESSTOKEN).
  --azdo-ref-name REF                  Ref name for template expansion (overrides AZDO_REFNAME).
  --azdo-timeout-seconds SECONDS       HTTP timeout override (overrides AZDO_TIMEOUT_SECONDS).
  --lint / --no-lint, -l / --no-l      Run yamllint (aliases: --lint, -l). All checks are opt-in; lint is disabled by default.
  --schema / --no-schema, -s / --no-s  Validate against Microsoft's published YAML schema (aliases: --schema, -s).
  --preview / --no-preview, -p / --no-p
                                       Call the Azure DevOps preview endpoint (aliases: --preview, -p).
  --fail-fast / --no-fail-fast         Stop immediately after the first file that fails validation.
  --exclude PATTERN                    Glob pattern or relative path to skip during scanning. Repeat the option to exclude multiple files/directories.
  --help                               Show this message and exit.

Output format

Every file gets one row with three columns (yamllint / schema / preview). A passing stage prints pass; the first error message is shown otherwise (plus a “(+N more)” suffix when applicable). Example:

┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ File                 ┃ yamllint ┃ schema ┃ preview                      ┃
┣━━━━━━━━━━━━━━━━━━━━━━╋━━━━━━━━━━╋━━━━━━━━╋━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┫
┃ workflows/ci.yml     ┃ pass     ┃ pass   ┃ pass                         ┃
┃ workflows/deploy.yml ┃ L3 C5: … ┃ pass   ┃ path not found (+2 more)     ┃
┗━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━┻━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Validated 2 file(s). Failures: 1.

Exit code is non-zero whenever any file fails or when the preview/schema endpoints error.

CI integration

Add a job that installs uv, exports AZDO_*, and runs the command. When running inside Azure Pipelines you can reuse $(System.AccessToken) and the current pipeline id:

- job: Validate
  pool:
    vmImage: ubuntu-latest
  steps:
    - task: UsePythonVersion@0
      inputs:
        versionSpec: '3.12'

    - script: |
        uv tool install azure-pipeline-validator
        azure-pipeline-validator --lint --schema --preview workflows/
      env:
        AZDO_ORG: $(System.TeamFoundationCollectionUri)
        AZDO_PROJECT: $(System.TeamProject)
        AZDO_PIPELINE_ID: $(System.DefinitionId)
        AZDO_PAT: $(System.AccessToken)
        AZDO_REFNAME: $(Build.SourceBranch)

The preview call runs with yamlOverride, so no build is queued.

Development workflow

Local development (inside this repo):

cd /path/to/azure-pipeline-validator
uv run azure-pipeline-validator --help

# Format and lint
uv run ruff format
uv run ruff check

# Run the test suite
uv run python -m pytest

pyproject.toml configures Ruff (line length 100, py313) and pytest/coverage. The tests include CLI help verification plus mock preview responses that mirror the real API payload captured from Azure DevOps.

Publishing the package

The package is published to PyPI automatically via GitHub Actions when a new tag is pushed using Trusted Publishing (OIDC) - no API tokens needed!

First-time Setup (Required Before First Publish)

Important: Trusted Publishing cannot create new projects on PyPI. You must create the project manually first:

1. Create the project on PyPI:

   • Go to https://pypi.org/manage/projects/
   • Click "Add new project"
   • Enter project name: azure-pipeline-validator (must match name in pyproject.toml)
     • Click "Create"

2. Set up Trusted Publishing:

   • Go to https://pypi.org/manage/account/publishing/
   • Click "Add a new trusted publisher"
   • Fill in:
     • PyPI project name: azure-pipeline-validator
     • Owner: andrewmaspero (your GitHub username)
     • Repository name: azure-pipeline-validator
     • Workflow filename: pipeline.yml
     • Environment name: pypi (optional but recommended)
   • Click "Add trusted publisher"

Publishing a New Version

1. Update version in pyproject.toml.
2. Commit and push the changes.
3. Create and push a tag: git tag v0.x.y && git push --tags.

The CI pipeline will automatically:

• Run all tests
• Build the package
• Publish to PyPI using Trusted Publishing (OIDC)

Once published, consumers can install and use it via:

# Using uvx (no installation needed)
uvx azure-pipeline-validator --help

# Or install globally
uv tool install azure-pipeline-validator
azure-pipeline-validator --help

# Or with pip
pip install azure-pipeline-validator
azure-pipeline-validator --help

For manual publishing, use uv directly:

uv build
uv publish

For manual publishing, you'll need to set UV_PUBLISH_USERNAME / UV_PUBLISH_PASSWORD environment variables, or use a PyPI API token.

Troubleshooting

Feel free to fork, contribute improvements, or publish your own build. This README should give you everything you need to adopt the validator in local workflows, UV-based tooling, and CI/CD.

License

azure-pipeline-validator is open source software released under the MIT License. Contributions are welcome—just open an issue or pull request so we can review changes together.