api-contract-tester 0.1.0

Production-grade API contract testing tool (schema-driven, pytest-friendly, CI-ready).

Intelligent API Contract Tester

Schema-driven API contract testing — Validate request/response payloads before they hit production.

---Invisible a fast way? Ecole's starlet Harry Potter.

🚀 Features

• JSON Schema Validation — Validate requests and responses against precise JSON Schema definitions
• Dual-Mode Testing — Mock (offline) or Live (real API) execution in the same framework
• GraphQL Support 🔴 NEW — Full GraphQL API contract testing (Phase 1 complete)
• OpenAPI Integration — Auto-generate contract tests from OpenAPI 3.x specifications
• Multi-Format Reporting — JSON (machine-readable) and HTML (human-friendly) reports
• Plugin Architecture — Extensible validator system for domain-specific rules (email validation, business logic checks)
• Schema Evolution Detection — Track API schema changes and detect breaking modifications
• Framework Discovery — Auto-detect endpoints in Python (Flask, FastAPI) and Node.js (Express) projects
• IDE Integration — VSCode extension for real-time contract violation diagnostics
• CLI & Programmatic APIs — Use as command-line tool or import as Python library
• Production-Oriented Design — Structured logging, configurable timeouts, basic error handling (improving)

---

🧠 Why This Project?

The Problem

API testing is fragmented. Teams either:

• Write brittle manual tests that break with schema changes
• Duplicate testing logic across frontend, backend, and integration suites
• Lack visibility into API contracts until integration breaks in production
• Struggle with environment-specific test setup (dev vs. staging vs. production)

The Solution

API Contract Tester centralizes API validation by making the schema the single source of truth:

OpenAPI Spec / JSON Schema
        ↓
    Contracts (YAML)
        ↓
   Mock + Live Tests (Single Run)
        ↓
   JSON + HTML Reports + IDE Diagnostics

This approach enables:

• ✅ CI/CD Integration — Fail builds when API contracts break
• ✅ Offline Testing — Test without a running server
• ✅ Drift Detection — Catch unexpected API changes
• ✅ Consumer-Driven Contracts — Formalize API agreements

⚠️ MVP Status: This project is an evolving MVP. Some advanced features (async execution, authentication, distributed testing) are planned for future releases.

---

🔥 GraphQL Support (NEW!)

The API Contract Tester now supports GraphQL API contract testing alongside REST APIs!

Quick Example

# contracts/graphql_get_user.yaml
name: graphql_get_user
api_type: graphql
endpoint: /graphql
method: POST
operation_type: query

query: |
  query GetUser($id: ID!) {
    user(id: $id) {
      id
      name
      email
    }
  }

variables:
  id: "123"

response_schema:
  data:
    type: object
    properties:
      user:
        type: object
        properties:
          id: { type: string }
          name: { type: string }
          email: { type: string }
        required: [id, name, email]
    required: [user]

GraphQL CLI Commands

# Parse and validate GraphQL schema
api-tester parse-graphql schema.graphql --verbose

# Introspect GraphQL endpoint
api-tester introspect http://localhost:4000/graphql --output schema.graphql

# Generate contracts from endpoint
api-tester generate-from-graphql http://localhost:4000/graphql

# Validate GraphQL query
api-tester validate-query query.graphql --schema schema.graphql

# Run GraphQL tests (mixed with REST)
api-tester run --mode live --base-url http://localhost:4000

📖 Full Documentation: GraphQL User Guide

---

🏗 Architecture

┌─────────────────────────────────────────┐
│            CLI Layer                    │
│   (init, run, report, scan, drift)      │
└──────────────┬──────────────────────────┘
               │
┌──────────────v──────────────────────────┐
│      Contract Runner                    │
│  (orchestrates test execution)          │
└──────────────┬──────────────────────────┘
               │
    ┌──────────┴──────────┐
    │                     │
┌───v──┐  ┌──────┐  ┌────v────┐
│Mock  │  │Live  │  │Reporters│
│Mode  │  │Mode  │  │(JSON,   │
│      │  │      │  │ HTML)   │
└──────┘  └──────┘  └─────────┘
    │         │
    └────┬────┘
         │
┌────────v─────────────────┐
│ Validators               │
│ • Schema (jsonschema)    │
│ • Request validation     │
│ • Custom validators      │
│ • Schema drift detector  │
└─────────────────────────┘
     │
┌────v─────────────────────┐
│ Integration Layer        │
│ • YAML/JSON loaders      │
│ • OpenAPI parser         │
│ • HTTP client (httpx)    │
│ • Framework adapters     │
└──────────────────────────┘

Design Principles:

• Modularity: Each component (loader, validator, reporter) is independent
• Extensibility: Plugin registry for custom validators without code changes
• Determinism: Same inputs always produce same outputs (ideal for CI/CD)
• Observability: Structured JSON logging and detailed error reporting

---

📦 Installation

From PyPI (Recommended)

pip install api-contract-tester

From Source

git clone https://github.com/yourusername/api-contract-tester.git
cd api-contract-tester
pip install -e .

Requirements

• Python 3.9+
• Pydantic 2.6+
• jsonschema 4.21+
• httpx 0.27+
• PyYAML 6.0+

---

⚙️ Configuration

Quick Start

api-tester init

This creates a sample project structure:

config/
  settings.yaml
contracts/
  users.yaml
  responses/
    get_users_200.json
    post_users_201.json
reports/

Settings (config/settings.yaml)

mode: mock                          # mock, live, or replay
base_url: https://api.example.com   # API endpoint (required for live mode)
contracts_dir: ./contracts          # Contract files location
responses_dir: ./contracts/responses # Mock response files
reports_dir: ./reports              # Output directory
baseline_dir: ./baseline_schemas    # Schema drift tracking
log_level: INFO
timeout_s: 10.0

Contract Format (YAML)

- name: get_users
  endpoint: /users
  method: GET
  expected_status: 200
  response_schema:
    type: array
    items:
      type: object
      properties:
        id: { type: integer }
        name: { type: string }
        email: { type: string, format: email }
      required: [id, name, email]
  custom_validators:
    - email_regex

- name: post_users
  endpoint: /users
  method: POST
  expected_status: 201
  request_schema:
    type: object
    properties:
      name: { type: string, minLength: 1 }
      email: { type: string, format: email }
    required: [name, email]
  response_schema:
    type: object
    properties:
      id: { type: integer }
      name: { type: string }
      email: { type: string, format: email }
    required: [id, name, email]
  examples:
    - name: valid_user
      request_payload:
        name: Jane Doe
        email: [e-mail]
      mock_response_file: responses/post_users_201.json

---

▶️ Usage

CLI Commands

1. Initialize Project

api-tester init --force

2. Run Contract Tests

# Mock mode (offline testing)
api-tester run --mode mock

# Live mode (against real API)
api-tester run --mode live --base-url https://api.production.com

# Custom output paths
api-tester run \
  --json-out ./results.json \
  --html-out ./report.html \
  --format human

3. Detect Schema Drift

api-tester detect-drift --mode live --base-url https://api.example.com

Compares current API responses against saved baseline schemas. Fails the build if breaking changes (removed fields) are detected.

4. Generate from OpenAPI

api-tester generate-from-openapi ./openapi.yaml --base-url https://api.example.com

Auto-generates contracts/*.yaml from OpenAPI 3.x specification.

5. Auto-Discover & Test

api-tester scan-and-test --base-url https://localhost:3000

Scans Python/Node.js source code for API endpoints, auto-generates contracts, and runs tests.

6. Generate Report

api-tester report --input ./results.json --output ./report.html

Programmatic Usage

from api_contract_tester.core.contract_runner import ContractRunner
from api_contract_tester.config.settings import Settings
from api_contract_tester.core.schema_loader import load_contracts_from_dir

settings = Settings(mode="mock", base_url="https://api.example.com")
contracts = load_contracts_from_dir("./contracts")

runner = ContractRunner(settings)
run_result = runner.run(contracts)

print(f"Passed: {run_result.passed}/{run_result.total}")
for result in run_result.results:
    if not result.passed:
        for violation in result.violations:
            print(f"  {violation.message} @ {violation.instance_path}")

---

📊 Example Output

HTML Report

Beautiful, responsive report generated automatically:

API Contract Test Report
─────────────────────────
Total Tests: 12
Passed: 10 ✓
Failed: 2 ✗
Duration: 125ms
Success Rate: 83.3%

Results:
┌─────────┬──────────────────────┬────────┬──────────┐
│ Method  │ Endpoint             │ Status │ Duration │
├─────────┼──────────────────────┼────────┼──────────┤
│ GET     │ /users               │ PASS   │ 45ms     │
│ POST    │ /users               │ PASS   │ 52ms     │
│ GET     │ /users/{id}          │ FAIL   │ 28ms     │
│ DELETE  │ /users/{id}          │ PASS   │ 35ms     │
│ ...     │ ...                  │ ...    │ ...      │
└─────────┴──────────────────────┴────────┴──────────┘

JSON Report

{
  "mode": "live",
  "base_url": "https://api.example.com",
  "total": 12,
  "passed": 10,
  "failed": 2,
  "duration_s": 0.125,
  "tests_per_sec": 96.0,
  "success_rate": 0.833,
  "results": [
    {
      "contract_name": "get_users",
      "method": "GET",
      "endpoint": "/users",
      "passed": true,
      "expected_status": 200,
      "actual_status": 200,
      "duration_ms": 45.2,
      "violations": []
    },
    {
      "contract_name": "get_user_by_id",
      "method": "GET",
      "endpoint": "/users/1",
      "passed": false,
      "expected_status": 200,
      "actual_status": 200,
      "duration_ms": 28.7,
      "violations": [
        {
          "kind": "response_schema",
          "message": "123 is not of type 'string'",
          "instance_path": "id",
          "schema_path": "properties.id.type"
        }
      ]
    }
  ]
}

---

🔌 Extensibility

Custom Validators

Define business-logic validators without modifying the core framework:

# api_contract_tester/validators/custom_rules.py
def _email_regex(_contract: Contract, response_body: Any) -> List[Violation]:
    """Validates that all 'email' fields match a valid email pattern."""
    email_re = re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$")
    violations = []

    def walk(x: Any, path: str = "") -> None:
        if isinstance(x, dict):
            for k, v in x.items():
                walk(v, f"{path}.{k}" if path else k)
        elif isinstance(x, list):
            for i, v in enumerate(x):
                walk(v, f"{path}[{i}]")
        else:
            if path.endswith("email") and isinstance(x, str):
                if not email_re.match(x):
                    violations.append(
                        Violation(
                            kind="custom_validation",
                            message="Invalid email format",
                            instance_path=path
                        )
                    )

    walk(response_body)
    return violations

_REGISTRY = {
    "email_regex": _email_regex,
}

Then reference in your contract:

custom_validators:
  - email_regex

Custom Reporters

Add new output formats by extending the reporter interface:

from api_contract_tester.core.models import RunResult
from pathlib import Path

def write_custom_report(run_result: RunResult, output_path: Path) -> None:
    """Export results to CSV, Prometheus, DataDog, etc."""
    output_path.write_text(
        "\n".join([
            f"{r.contract_name},{r.method},{r.endpoint},"
            f"{r.passed},{r.duration_ms}"
            for r in run_result.results
        ])
    )

Framework Adapters

Extend endpoint discovery for additional frameworks:

from api_contract_tester.core.framework_adapters.base import FrameworkAdapter
from api_contract_tester.core.models import Contract

class GoFrameworkAdapter(FrameworkAdapter):
    def discover_endpoints(self, root: Path) -> List[Contract]:
        """Scan Go source files for Gin/Echo route definitions."""
        # Implementation here
        pass

---

🧪 Testing

Run Tests

# All tests (with automatic coverage)
pytest tests/

# Specific test file
pytest tests/test_contract_runner.py

# With coverage (manual)
pytest --cov=api_contract_tester --cov-report=html --cov-report=xml

# Run all tests with coverage reporting
api-tester coverage

# Coverage with custom threshold
api-tester coverage --fail-under 95

# Coverage for specific tests only
api-tester coverage tests/test_contract_runner.py

Coverage Reporting

The project uses coverage.py to measure code coverage with a 90% minimum threshold.

Generate Coverage Reports

# Run tests with coverage (HTML + XML + terminal)
api-tester coverage

# Generate all report formats including JSON
api-tester coverage --json

# Skip terminal output (CI/CD friendly)
api-tester coverage --no-term

# Only generate HTML report
api-tester coverage --no-xml --no-term

Coverage Report Formats

• HTML Report (htmlcov/index.html): Interactive browser-based report with syntax highlighting
• XML Report (coverage.xml): For CI/CD integration (Codecov, SonarQube)
• JSON Report (coverage.json): Machine-readable format for custom tooling
• Terminal Report: Immediate feedback in console with missing lines

View Coverage Reports

# Open HTML report in browser
start htmlcov/index.html  # Windows
open htmlcov/index.html   # macOS
xdg-open htmlcov/index.html  # Linux

Test Coverage

• Core validation: 90%+
• CLI commands: 90%+
• Framework adapters: 90%+
• Error handling: 90%+
• Overall target: 90% minimum

Example Test

def test_schema_validation_detects_type_mismatch():
    schema = {
        "type": "object",
        "properties": {"id": {"type": "integer"}},
        "required": ["id"]
    }
    instance = {"id": "not-an-integer"}

    violations = validate_json_schema(instance, schema)

    assert len(violations) == 1
    assert "is not of type" in violations[0].message

---

📈 Performance & Scalability

Benchmarks

10 contracts (mock mode):     ~5ms
100 contracts (mock mode):    ~50ms
10 contracts (live, 100ms latency): ~1.0s
1000 contracts (mock mode):   ~500ms

Optimization Notes

• ✅ Single-threaded execution (simple, deterministic)
• ⚠️ For 10k+ contracts, consider async/await refactor
• ✅ Mock mode is I/O bound; Live mode is network bound
• ✅ Schema validators are cached per schema
• 🚀 Roadmap: Async execution, worker pools, distributed testing

Memory Usage

Small project (10 contracts):    ~5MB
Medium project (100 contracts):  ~25MB
Large project (1000 contracts):  ~200MB

---

🛣 Roadmap

v0.2.0 (Q2 2024) - Current Release Candidate

• Async/concurrent test execution
• Response header validation
• Contract versioning system
• Docker containerization
• Prometheus metrics
• API mock server
• Retry logic & error recovery
• Coverage.py integration
• Request payload templating ({{variable}} support)
• CI/CD pipeline templates (GitHub Actions, GitLab CI)
• WebSocket endpoint support

v0.3.0 (Q3 2024) - Production Requirement Release

• GraphQL Support (🔴 REQUIRED FOR PRODUCTION)
  • GraphQL schema validation
  • Query/mutation testing
  • Introspection support
  • GraphQL contract format
  • Schema drift detection for GraphQL
• OAuth2/JWT authentication helpers
• Database fixture seeding & teardown
• GraphQL security audit

v0.4.0+ (Q4 2024+)

• Distributed test execution (worker pools, message queues)
• Performance profiling & load testing
• gRPC contract testing
• Postman collection integration
• CloudWatch/DataDog integration

---

⚠️ Production Deployment Notice

Current Status (v0.2.0-rc1)

✅ REST API Support: Fully implemented and production-ready ❌ GraphQL Support: Not yet implemented - REQUIRED for production

What This Means

You CAN use this project for:

• Development and testing of REST APIs
• Staging environment validation
• CI/CD pipeline integration (REST-only)
• Early production trials (REST APIs only)

You CANNOT use this project for:

• Full production deployment (until GraphQL is added)
• GraphQL API contract testing (not yet supported)
• Enterprise environments requiring GraphQL
• v1.0.0 General Availability (GraphQL is mandatory)

Timeline for GraphQL Support

Milestone	Date	Status
Design & Planning	April 2026	✅ Complete
Implementation	Q3 2026	🎯 Planned
Beta Testing	Q4 2026	📅 Scheduled
Production Ready	Q1 2027	📅 Scheduled

See GraphQL Production Requirement for details.

---

🤝 Contributing

We welcome contributions! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Test your changes (pytest tests/)
4. Format your code (black api_contract_tester/)
5. Commit with clear messages (git commit -m 'Add amazing feature')
6. Push to the branch (git push origin feature/amazing-feature)
7. Open a Pull Request

Development Setup

git clone https://github.com/yourusername/api-contract-tester.git
cd api-contract-tester
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
pytest tests/

Code Standards

• Python: 3.9+ only
• Type hints: Required on all public functions
• Tests: 80%+ coverage minimum
• Docstrings: Google-style format
• Formatting: Black, 100 character line limit

---

📚 Documentation

• Getting Started — Step-by-step setup guide
• API Reference — Python API documentation
• Examples — Real-world project examples
• Schema Writing Guide — How to write contracts
• FAQ — Common questions

---

📄 License

MIT License — See LICENSE file for details.

---

🙋 Support

• GitHub Issues: Report bugs or request features
• Discussions: Q&A and ideas
• Email: [e-mail]

---

💡 Acknowledgments

Built on solid foundations:

• jsonschema — JSON Schema validation
• Pydantic — Data validation
• httpx — Async HTTP client
• prance — OpenAPI utilities

---

Made with ❤️ by developers, for developers.

⭐ If you find this useful, please star the repository!