Swagger Coverage Tool

The Swagger Coverage Tool is designed to measure API test coverage based on Swagger documentation. It provides automated tracking and reporting of test coverage for APIs, helping ensure that your endpoints and services are well-tested.

Features

• Automatic Coverage Measurement: The tool automatically measures coverage, generating a report based on the collected data.
• Multiple Swagger Documentations: It supports working with multiple Swagger documentation files. For example, if you have multiple microservices with their own Swagger documentation, the coverage can be measured separately for each service.
• Overall Service Coverage: It calculates the total coverage of each service.
• Endpoint Coverage: Measures whether each endpoint is covered by tests and to what percentage. It also counts the number of test cases that cover a given endpoint.
• Status Code Coverage: Tracks which status codes were covered, including the number of test cases that cover each status code.
• Request and response coverage: The tool checks whether a request or response exists for a given endpoint, and if so, verifies whether it was covered.
• Query parameter coverage: The tool checks whether the query parameters defined in the Swagger documentation were covered for each endpoint.
• History of Coverage: Maintains a history of coverage for each service and endpoint.
• Flexible Searching and Sorting: Supports flexible data search and sorting, making it easy to analyze coverage results.
• HTML Report Generation: All coverage data is aggregated into a single index.html report file, which can be opened, shared, or published.
• Support for httpx and requests Libraries: The tool works with both httpx and requests libraries for making HTTP requests.

Links

Example Report

You can view an example of a coverage report generated by the tool here.

Questions & Support

If you have any questions or need assistance, feel free to ask @Nikita Filonov.

Preview

Summary

Endpoints

Details

Installation

pip install swagger-coverage-tool

Usage

Simple Example with httpx

Here's an example of how to use the tool with httpx:

import httpx

from swagger_coverage_tool import SwaggerCoverageTracker

# Initialize the tracker with service
tracker = SwaggerCoverageTracker(service="api-service")


# Track coverage for the "get_user" endpoint
@tracker.track_coverage_httpx("/api/v1/users/{user_id}")
def get_user(user_id: str):
    return httpx.get(f"http://localhost:8000/api/v1/users/{user_id}")


# Track coverage for the "create_user" endpoint
@tracker.track_coverage_httpx("/api/v1/users")
def create_user():
    return httpx.post("http://localhost:8000/api/v1/users")


# Make requests
get_user("123")
create_user()

After executing the HTTP requests, coverage data will be automatically collected and saved to the coverage-results folder by default.

Simple Example with requests

Here's the same example using the requests library:

import requests

from swagger_coverage_tool import SwaggerCoverageTracker

tracker = SwaggerCoverageTracker(service="api-service")


@tracker.track_coverage_requests("/api/v1/users/{user_id}")
def get_user(user_id: str) -> requests.Response:
    return requests.get(f"http://localhost:8000/api/v1/users/{user_id}")


@tracker.track_coverage_requests("/api/v1/users")
def create_user() -> requests.Response:
    return requests.post("http://localhost:8000/api/v1/users")


get_user()
create_user()

Coverage Report Generation

After each tracked HTTP request, the tool stores coverage data as JSON files in ./coverage-results/ by default. The folder is created automatically, and its files remain there between test runs.

Once your tests are complete, generate a detailed coverage report with:

swagger-coverage-tool save-report

save-report includes every coverage result in the configured results_dir, including files from earlier test runs.

This will generate:

• index.html — a standalone HTML report that you can open, share, or publish.
• coverage-report.json — a structured JSON report for CI/CD summaries and custom integrations.

Run swagger-coverage-tool save-report from the directory containing your configuration files, or set custom config paths via SWAGGER_COVERAGE_CONFIG_FILE_*.

Configuration

You can configure the Swagger Coverage Tool using a single file: either a YAML, JSON, or .env file. By default, the tool looks for configuration in:

• swagger_coverage_config.yaml
• swagger_coverage_config.json
• .env (for environment variable configuration)

By default, these files are loaded from the current working directory. Configuration is automatically loaded via get_settings().

Override config file paths

You can override default config locations using environment variables:

• SWAGGER_COVERAGE_CONFIG_FILE_YAML — path to YAML config
• SWAGGER_COVERAGE_CONFIG_FILE_JSON — path to JSON config
• SWAGGER_COVERAGE_CONFIG_FILE_ENV — path to .env config

Example:

SWAGGER_COVERAGE_CONFIG_FILE_YAML=./ci/swagger_coverage_config.yaml swagger-coverage-tool save-report

Configuration via .env

All settings can be declared using environment variables. Nested fields use dot notation, and all variables must be prefixed with SWAGGER_COVERAGE_.

Example: .env

# Define the services that should be tracked. In the case of multiple services, they can be added in a comma-separated list.
SWAGGER_COVERAGE_SERVICES='[
    {
        "key": "my-api-service",
        "name": "My API Service",
        "tags": ["API", "PRODUCTION"],
        "repository": "https://github.com/my-api",
        "swagger_url": "https://my-api.com/swagger.json"
    }
]'

# The directory where the coverage results will be saved.
SWAGGER_COVERAGE_RESULTS_DIR="./coverage-results"

# The file that stores the history of coverage results.
SWAGGER_COVERAGE_HISTORY_FILE="./coverage-history.json"

# The retention limit for the coverage history. It controls how many historical results to keep.
SWAGGER_COVERAGE_HISTORY_RETENTION_LIMIT=30

# Optional file paths for the HTML and JSON reports.
SWAGGER_COVERAGE_HTML_REPORT_FILE="./index.html"
SWAGGER_COVERAGE_JSON_REPORT_FILE="./coverage-report.json"

Note: Either swagger_url or swagger_file is required for each service.

Configuration via YAML

Example: swagger_coverage_config.yaml

services:
  - key: "my-api-service"
    name: "My API Service"
    tags: [ "API", "PRODUCTION" ]
    repository: "https://github.com/my-api"
    swagger_url: "https://my-api.com/swagger.json"
    # swagger_file: "swagger_file_path.json"  # Optional if not using swagger_url

results_dir: "./coverage-results"
history_file: "./coverage-history.json"
history_retention_limit: 30
html_report_file: "./index.html"
json_report_file: "./coverage-report.json"

Configuration via JSON

Example: swagger_coverage_config.json

{
  "services": [
    {
      "key": "my-api-service",
      "name": "My API Service",
      "tags": [
        "API",
        "PRODUCTION"
      ],
      "repository": "https://github.com/my-api",
      "swagger_url": "https://my-api.com/swagger.json"
    }
  ],
  "results_dir": "./coverage-results",
  "history_file": "./coverage-history.json",
  "history_retention_limit": 30,
  "html_report_file": "./index.html",
  "json_report_file": "./coverage-report.json"
}

Configuration Reference

How It Works

Once configured, the tool automatically:

• Loads all Swagger definitions from the given URLs or files.
• Tracks test coverage during API calls.
• Writes raw coverage data to coverage-results/.
• Stores optional historical data and generates an HTML report at the end.

Command-Line Interface (CLI)

The Swagger Coverage Tool provides several CLI commands to help with managing and generating coverage reports.

Command: save-report

Generates a detailed coverage report based on the collected result files. This command will process all the raw coverage data stored in the coverage-results directory and generate an HTML report.

Usage:

swagger-coverage-tool save-report

• This is the main command to generate a coverage report. After executing API tests and collecting coverage data, use this command to aggregate the results into a final report.
• The report is saved as an HTML file, typically named index.html, which can be opened in any browser.

Command: copy-report

This is an internal command mainly used during local development. It updates the report template for the generated coverage reports. It is typically used to ensure that the latest report template is available when you generate new reports.

Usage:

swagger-coverage-tool copy-report

• This command updates the internal template used by the save-report command. It's useful if the template structure or styling has changed and you need the latest version for your reports.
• This command is typically only used by developers working on the tool itself.

Command: print-config

Prints the resolved configuration to the console. This can be useful for debugging or verifying that the configuration file has been loaded and parsed correctly.

Usage:

swagger-coverage-tool print-config

• This command reads the configuration file (swagger_coverage_config.yaml, swagger_coverage_config.json, or .env, or a custom path set via SWAGGER_COVERAGE_CONFIG_FILE_*) and prints the final configuration values to the console.
• It helps verify that the correct settings are being applied and is particularly useful if something is not working as expected.

Command: clear-results

Removes JSON files from the configured results_dir. Run it before starting a new test run when the next report should include only that run. Keep results_dir dedicated to coverage results: all JSON files there are removed except configured history and report files. Non-JSON files and nested directories are preserved.

Usage:

swagger-coverage-tool clear-results

Coverage history is preserved, so reports can show trends across test runs while counting only the current run's cases. The command succeeds if results_dir does not exist and exits with an error if files cannot be removed.

Troubleshooting

The report is empty or missing data

• Ensure that your HTTP client methods use track_coverage_httpx() or track_coverage_requests().
• Match the tracked endpoint name and HTTP method to the Swagger schema, including path placeholders.
• Run swagger-coverage-tool save-report from the directory containing your configuration, or set custom config paths.
• Use swagger-coverage-tool print-config to check the resolved settings.
• Check that the configured results_dir contains .json files from your tests.

The report includes cases from earlier test runs

Run swagger-coverage-tool clear-results before starting a new test run, then execute your tests and run swagger-coverage-tool save-report.