SpecSentinel

Your OpenAPI spec says one thing. Your API does another. SpecSentinel calls the running API, compares every answer with the spec and tells you where they disagree.

It reports three kinds of drift:

• missing fields
• wrong data types
• undocumented status codes

Exit code 0 means spec and API match. Exit code 1 means drift. Exit code 2 means the check could not be completed. That makes it a one line gate in a CI/CD pipeline.

New in 0.2.0

• More warnings for the values an API returns: string formats (date-time, uuid, email, uri), string lengths, numeric ranges and patterns, and required response headers that are missing.
• --format json now adds per severity counts and a flat findings list next to the per operation details.
• The GitHub Action takes a version input to install an exact SpecSentinel release instead of the latest one.

Note for --strict users: the new checks are warnings, so a run that used to be green can turn red with --strict. Review the new findings before enabling strict in an existing pipeline.

Install

Python 3.9 or newer.

pip install specsentinel

Or from a clone of this repository:

pip install .

Try it in one minute

The repository ships a demo API that drifts from its spec on purpose.

python examples/demo_server.py &
specsentinel examples/petstore.yaml --url http://127.0.0.1:8099

Real output:

SpecSentinel 0.2.0
Spec:   examples/petstore.yaml
Target: http://127.0.0.1:8099

GET /health            200  OK
GET /pets              200  OK
GET /pets/{petId}      200  DRIFT
    error   MISSING_FIELD            body.name
            required field is missing in the response
    error   TYPE_MISMATCH            body.id
            expected integer, got string
    warning UNDOCUMENTED_FIELD       body.breed
            field is returned but not described in the spec
GET /stats             503  DRIFT
    error   UNDOCUMENTED_STATUS      status
            status 503 is not documented (documented: 200)
GET /owners/{ownerId}  -    SKIPPED
    no example value for path parameter 'ownerId' (pass --param ownerId=VALUE)

4 checked, 2 with drift, 1 skipped, 0 failed
Result: DRIFT (exit code 1)

Start the demo with --conform and the same command ends with Result: MATCH (exit code 0).

Exit codes

Code 2 is deliberately not 0. A pipeline should never turn green because nothing was actually checked.

What counts as drift

Warnings do not fail the run. Add --strict and they do.

Options

specsentinel SPEC --url BASE_URL [options]

  SPEC                   path or URL of the OpenAPI 3.x document, YAML or JSON
  --url BASE_URL         base URL of the running API
  -H, --header 'N: v'    header sent with every request, repeatable
  --param NAME=VALUE     value for a path or query parameter, repeatable
  --params-file FILE     YAML or JSON file with parameter values
  --timeout SECONDS      wait per request, default 10
  --strict               treat warnings as drift
  --format text|json     output format, default text
  --version

Use --format json when another tool should read the result. The output is a single JSON document with exit_code, a summary with the operation counts and counts per severity (error/warning), a flat findings list (each entry is code, method, path and severity) and one operations entry per checked operation with its detailed findings. Header values are never included in the output.

{
  "version": "0.2.0",
  "spec": "examples/petstore.yaml",
  "target": "http://127.0.0.1:8099",
  "strict": false,
  "exit_code": 1,
  "summary": {
    "checked": 4,
    "drift": 1,
    "skipped": 1,
    "failed": 0,
    "counts": {"error": 1, "warning": 1}
  },
  "findings": [
    {"code": "MISSING_FIELD", "method": "GET", "path": "/pets/{petId}", "severity": "error"},
    {"code": "UNDOCUMENTED_FIELD", "method": "GET", "path": "/pets/{petId}", "severity": "warning"}
  ],
  "operations": [
    {
      "method": "GET",
      "path": "/pets/{petId}",
      "status": 200,
      "state": "DRIFT",
      "findings": [
        {"severity": "error", "code": "MISSING_FIELD", "location": "body.name", "message": "required field is missing in the response"},
        {"severity": "warning", "code": "UNDOCUMENTED_FIELD", "location": "body.extra", "message": "field is not documented in the spec"}
      ]
    }
  ]
}

How requests are built

SpecSentinel sends GET requests only. Methods that change data could damage the API under test, so they are out of scope by design.

Parameter values come from the spec: example, examples, the schema example, default, or the first enum value. A path parameter without any of these cannot be filled in automatically. That operation is reported as SKIPPED with the exact --param to pass. Skipped operations do not fail the run.

Params file

Large APIs have many operations with path parameters. Put the values in a file instead of a long command line:

# params.yaml
petId: 1                 # used by every operation with a parameter called petId
ownerId: 7
GET /pets/{petId}:       # applies to this one operation only
  petId: 42

specsentinel openapi.yaml --url https://staging.example.com --params-file params.yaml

Order of precedence, later wins: spec example, params file, params file entry for one operation, --param on the command line.

For authentication pass the header yourself:

specsentinel openapi.yaml --url https://staging.example.com \
  --header "Authorization: Bearer $API_TOKEN"

GitHub Actions

name: API contract
on: [push, pull_request]

jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: b6bs62fhys-jpg/specsentinel@v0.2.0
        with:
          spec: openapi.yaml
          url: https://staging.example.com
          header: "Authorization: Bearer ${{ secrets.API_TOKEN }}"

The action installs SpecSentinel from PyPI, runs it against your API and passes its exit code through unchanged: 0 on match, 1 on drift, 2 if the check could not be completed. When the API drifts, exit code 1 fails the step and the build goes red.

Inputs: spec (path or URL) and url are required. Optional inputs are header (sent with every request, passed through an environment variable and not written to the log), params_file, strict (default false), python_version (default 3.12) and version. version pins the exact SpecSentinel release that gets installed, for example 0.2.0. When it is empty, the latest release is installed.

Tested against real specifications

The schema checks were run against the public OpenAPI descriptions of Swagger Petstore, GitHub and Stripe: 920 GET operations and 2032 documented JSON responses in total. For every response a conforming example was generated from its schema and fed through the checker. It produced no findings and no crashes.

You can repeat the run yourself with python tools/spec_smoke.py <spec file or URL>. The exact output is in docs/smoke_results.md.

The generator and the checker share the same reading of the schema, so this run shows that the checker raises no false alarms on large real specifications. It does not show that every kind of drift is caught, and it does not replace running SpecSentinel against your own live API.

Found in the wild

On 2026-09-21 SpecSentinel 0.1.1 was run against the public Swagger Petstore. It reported one drift: GET /user/login is documented as JSON and answers with Content-Type: application/json, but the body is plain text. Three endpoints answered 500 and were reported as SERVER_ERROR warnings. The Petstore is a shared demo server, so results may differ. The full output is in docs/live_petstore.txt.

Security

SpecSentinel follows $ref across files and URLs. A spec can make SpecSentinel read files from its own machine and fetch documents from any URL it can reach. Only check specs you trust: review a spec before running it, and only point the tool at specs from sources you control or rely on, just as you would with any code you choose to execute.

Limits of this version

This is an early release. Known limits:

• GET operations only
• JSON response bodies only, other content types are not compared
• allOf is merged, oneOf and anyOf pass when any alternative fits

Feedback on which check should come next is very welcome. Open an issue.

Development

pip install -e ".[dev]"
pytest

License

MIT, see LICENSE.