PyPlayKit - Enterprise Test Automation Framework

Tagline: A modular, scalable, and CI/CD-ready Python + Playwright framework for enterprise automation supporting multiple projects and 50+ QE engineers.

Overview

PyPlayKit is a multi-project enterprise test automation framework built with Python, Playwright, and Pytest. It supports multiple independent projects with UI, API, Database, and Integration testing capabilities.

Key Features:

• 🏢 Multi-Project Support: Separate test organization for multiple applications
• 👥 Scalable: Designed for 50-100 QE engineers working simultaneously
• 🎯 Zero Merge Conflicts: Project-first organization eliminates conflicts
• 📊 Data Comparison: File-to-File, File-to-DB, DB-to-File validation (Excel, CSV, JSON)
• 📈 Interactive HTML Reports: Filterable validation reports with KPIs ⭐ NEW
• 📚 Comprehensive Documentation: 8,700+ lines across 15+ guides
• 🔧 Extensible: Plugin architecture with orchestration support
• 🔍 Self-Healing: Locator recovery with fallback chains
• 🔭 Observable: Comprehensive metrics and reporting

Quick Start

Prerequisites

• Python 3.11 or higher
• pip

Setup (5 minutes)

# Windows PowerShell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
playwright install
pip install -r requirements-dev.txt

Set Environment Variables

# Required for tests
$env:PYPLAYKIT_TEST_USERNAME="standard_user"
$env:PYPLAYKIT_TEST_PASSWORD="secret_sauce"

Run Tests

# Run all smoke tests
pytest -m smoke

# Run specific project tests
pytest tests/projects/project_1/
pytest tests/projects/project_2/
pytest tests/projects/project_3/

# Run by marker
pytest -m project_1
pytest -m project_2
pytest -m project_3

Architecture

Multi-Project Structure

PyPlayKit supports multiple independent projects with separate test suites:

tests/
├── projects/                              # Multi-project organization
│   ├── project_1/                      # Project 1: Quote management
│   │   ├── functional/
│   │   │   ├── ui/                       # UI tests
│   │   │   ├── api/                      # API tests
│   │   │   └── database/                 # Database tests
│   │   └── integration/                  # Integration tests
│   │
│   ├── project_2/                        # Project 2: Dispatch & logistics
│   │   ├── functional/
│   │   │   ├── ui/
│   │   │   ├── api/
│   │   │   └── database/
│   │   └── integration/
│   │
│   └── project_3/            # Project 3: Customs management
│       ├── functional/
│       │   ├── ui/
│       │   ├── api/
│       │   └── database/
│       └── integration/
│
└── unit/                                  # Framework unit tests

Layered Architecture

1. Test Layer (tests/) - Multi-project organization with Pytest
2. Page Object Layer (pages/) - Encapsulated locators per project
3. Core Engine Layer (core/) - Playwright lifecycle and browser management
4. Utilities Layer (utils/) - Logging, data loading, assertions, API clients
5. Configuration Layer (config/) - Framework + project-specific configs
6. Plugin Layer (plugins/) - Extensible plugin architecture
7. Orchestration Layer (orchestration/) - Dependency-aware execution
8. Resilience Layer (resilience/) - Self-healing locator resolution
9. Integration Layer (integrations/) - Jira and test management adapters

Architecture Diagrams

Architecture Overview — the full layered dependency stack, cross-cutting layers (plugins, orchestration, resilience), the MCP read-only sidecar, and the security & governance rail:

Data Comparison Flow — config-driven Excel / CSV / DB comparison: sources → loader → comparison engine → result + HTML report:

Security Posture — defense-in-depth: runtime secrets/masking/auth controls plus the bandit / pip-audit / gitleaks CI scanning pipeline:

Diagrams are hosted in the public ShansDocsForPublicUsage repository so they render on the PyPI page. The source SVGs and the full architecture reference (docs/ARCHITECTURE.md — 9 Mermaid views covering layers, runtime lifecycle, config resolution, data engineering, plugins/observability, MCP, and security) live in the project's docs/ tree; regenerate the PNGs after editing an SVG with python scripts/svg_to_png.py.

Project Structure

pyplaykit/
├── config/
│   ├── config.yaml                       # Framework configuration
│   ├── environments.yaml                 # Global environments
│   └── projects/                         # Project-specific configs
│       ├── project_1.yaml
│       ├── project_2.yaml
│       └── project_3.yaml
│
├── core/
│   ├── base_page.py
│   ├── base_test.py
│   ├── browser_factory.py
│   └── playwright_manager.py
│
├── integrations/
│   └── adapters.py
│
├── orchestration/
│   └── planner.py
│
├── plugins/
│   ├── base.py
│   ├── registry.py
│   ├── api_plugin.py
│   ├── data_plugin.py
│   └── security_plugin.py
│
├── resilience/
│   └── locator_recovery.py
│
├── pages/                                # Page objects per project
│   ├── project_1/
│   ├── project_2/
│   └── project_3/
│
├── tests/
│   ├── projects/                         # All project tests
│   │   ├── project_1/
│   │   ├── project_2/
│   │   └── project_3/
│   └── unit/                             # Framework unit tests
│
├── utils/
│   ├── logger.py
│   ├── data_loader.py
│   ├── assertion_helper.py
│   ├── api_client.py
│   ├── response_validator.py
│   ├── data_validator.py
│   ├── config_reader.py
│   ├── observability.py
│   ├── environment_validator.py
│   └── tdm.py
│
├── test_data/                            # Test data per project
│   ├── project_1/
│   ├── project_2/
│   └── project_3/
│
├── reports/                              # Test reports and artifacts
│
├── conftest.py                           # Pytest configuration
├── pytest.ini                            # Pytest settings
├── pyproject.toml                        # Package metadata
├── requirements.txt                      # Dependencies
└── README.md                             # This file

Current Projects

1. project_1

Description: Quote management and generation system
Tests: tests/projects/project_1/
Config: config/projects/project_1.yaml
Marker: @pytest.mark.project_1

Run tests:

pytest tests/projects/project_1/
pytest -m project_1

2. project_2

Description: Dispatch and logistics management system
Tests: tests/projects/project_2/
Config: config/projects/project_2.yaml
Marker: @pytest.mark.project_2

Run tests:

pytest tests/projects/project_2/
pytest -m project_2

3. Customs Modernization

Description: Customs management system modernization
Tests: tests/projects/project_3/
Config: config/projects/project_3.yaml
Marker: @pytest.mark.project_3

Run tests:

pytest tests/projects/project_3/
pytest -m project_3

Running Tests

By Project

# project_1
pytest tests/projects/project_1/
pytest tests/projects/project_1/ -m smoke

# project_2
pytest tests/projects/project_2/
pytest tests/projects/project_2/ -m smoke

# Customs Modernization
pytest tests/projects/project_3/
pytest tests/projects/project_3/ -m smoke

By Test Type

# UI tests only
pytest tests/projects/project_1/functional/ui/

# API tests only
pytest tests/projects/project_2/functional/api/

# Database tests only
pytest tests/projects/project_3/functional/database/

# Integration tests only
pytest tests/projects/project_1/integration/

By Marker

# Project markers
pytest -m project_1
pytest -m project_2
pytest -m project_3

# Test type markers
pytest -m api
pytest -m ui
pytest -m database
pytest -m integration

# Combined markers
pytest -m "project_1 and smoke"
pytest -m "project_2 and api"

All Projects

# Run smoke tests for all projects
pytest tests/projects/ -m smoke

# Run all tests for all projects
pytest tests/projects/

# Run specific test type across all projects
pytest tests/projects/ -m api
pytest tests/projects/ -m ui

With Options

# Different environment
pytest tests/projects/project_1/ --pyplaykit-env qa
pytest tests/projects/project_2/ --pyplaykit-env uat

# Different browser
pytest tests/projects/project_1/ --pyplaykit-browser firefox --pyplaykit-headed

# Parallel execution
pytest tests/projects/project_1/ -n 4

# With readiness checks
pytest tests/projects/project_1/ --pyplaykit-readiness-check

Available pytest CLI Options

Registered in conftest.py:

• --pyplaykit-env — target environment (dev, qa, uat, prod)
• --pyplaykit-browser — browser (chromium, firefox, webkit)
• --pyplaykit-headed — disable headless mode
• --pyplaykit-base-url — override base URL
• --pyplaykit-readiness-check — enable environment readiness checks

Test Markers

Defined in pytest.ini:

Suite Markers

• @pytest.mark.smoke - Critical path tests
• @pytest.mark.sanity - Quick validation tests
• @pytest.mark.regression - Full regression suite

Project Markers

• @pytest.mark.project_1 - project_1 tests
• @pytest.mark.project_2 - project_2 tests
• @pytest.mark.project_3 - Customs Modernization tests

Test Type Markers

• @pytest.mark.ui - UI functional tests
• @pytest.mark.api - API functional tests
• @pytest.mark.database - Database validation tests
• @pytest.mark.integration - Integration tests

Feature Markers

• @pytest.mark.login - Login functionality
• @pytest.mark.quotes - Quote management
• @pytest.mark.logistics - Logistics/dispatch
• @pytest.mark.customs_mgmt - Customs management

Adding Tests

Where to Add Your Tests

Example: Adding a New Test

1. Create page object (for UI tests):

# pages/project_1/quote_creation_page.py
from core.base_page import BasePage

class QuoteCreationPage(BasePage):
    CUSTOMER_SELECT = "#customer"
    SAVE_BUTTON = "#save"
    
    def create_quote(self, customer: str):
        self.click(self.CUSTOMER_SELECT)
        self.click(f"option:has-text('{customer}')")
        self.click(self.SAVE_BUTTON)

2. Create test:

# tests/projects/project_1/functional/ui/quotes/test_create_quote.py
import pytest
from pages.project_1.quote_creation_page import QuoteCreationPage

@pytest.mark.project_1
@pytest.mark.smoke
@pytest.mark.quotes
def test_create_quote_with_valid_customer(page, runtime_options):
    quote_page = QuoteCreationPage(page, runtime_options["resilience_policy"])
    quote_page.navigate(runtime_options["base_url"] + "/quotes/new")
    quote_page.create_quote("ACME Corp")
    assert quote_page.is_quote_saved()

3. Run your test:

pytest tests/projects/project_1/functional/ui/quotes/test_create_quote.py -v

Data Comparison Patterns

PyPlayKit provides comprehensive data comparison capabilities using the built-in DataValidator utility.

Prerequisites for Data Comparison

# Option 1: Install with optional dependencies
pip install pyplaykit[data-comparison]

# Option 2: Install dependencies separately
pip install pandas openpyxl

# Option 3: Use the examples requirements file
pip install -r demo/requirements.txt

Note: Data comparison features require pandas and openpyxl. The framework will work without them for UI/API/Database testing.

Supported Comparison Types

Quick Example: Excel-to-Excel Comparison

Simple One-Line API (Recommended for QE Engineers):

import pytest
from utils.data_comparison_utils import compare_excel_files

@pytest.mark.data
def test_compare_excel_files(logger):
    # ONE function call - framework handles everything!
    result = compare_excel_files(
        source_file="report_baseline.xlsx",
        target_file="report_current.xlsx",
        float_tolerance=0.01
    )
    
    # Check results
    logger.info(f"{result.summary}")
    logger.info(f"HTML Report: {result.report_path}")
    
    if not result.passed:
        pytest.fail(f"Validation failed! {result.failed_count} discrepancies found.")

Manual Validation (For custom logic):

import pytest
import pandas as pd
from pyplaykit import DataValidator

@pytest.mark.data
def test_compare_with_custom_logic(logger):
    df1 = pd.read_excel("baseline.xlsx")
    df2 = pd.read_excel("current.xlsx")
    
    records1 = df1.to_dict('records')
    records2 = df2.to_dict('records')
    
    for idx, (r1, r2) in enumerate(zip(records1, records2)):
        DataValidator.assert_records_equal(r1, r2)
    
    logger.info("✓ Files match!")

DataValidator Methods

Runnable Examples

See demo/functional/database/data_comparison_examples.py for 8 complete examples:

# Run all data comparison examples
pytest demo/functional/database/data_comparison_examples.py -v -s

# Run specific example
pytest demo/functional/database/data_comparison_examples.py::test_excel_to_excel_basic -v -s

Interactive HTML Reports ⭐ NEW

Option 1: Automatic Report Generation (Recommended):

from utils.data_comparison_utils import compare_excel_files

# Framework automatically tracks ALL mismatches and generates HTML report
result = compare_excel_files(
    source_file="baseline.xlsx",
    target_file="current.xlsx",
    float_tolerance=0.01
)

# Result includes:
print(f"Passed: {result.passed}")
print(f"Matched: {result.passed_count}")
print(f"Failed: {result.failed_count}")
print(f"Total Mismatches: {result.total_mismatches}")
print(f"Report: {result.report_path}")

Available Functions:

File-to-File:

• compare_excel_files() - Excel to Excel
• compare_csv_files() - CSV to CSV

File-to-Database:

• compare_excel_to_db() - Excel to Database
• compare_csv_to_db() - CSV to Database

Database-to-File:

• compare_db_to_excel() - Database to Excel
• compare_db_to_csv() - Database to CSV

Database-to-Database:

• compare_db_to_db() - Database to Database

Advanced:

• compare_dataframes() - DataFrame to DataFrame (custom sources)

Option 2: Manual Report Building (Advanced):

from utils.data_comparison_report import DataComparisonReport

# For custom comparison logic
report = DataComparisonReport()
report.set_comparison_type("FILE_TO_FILE (Excel)")
report.set_source("baseline.xlsx", 100)
report.set_target("current.xlsx", 100)

# Your custom comparison logic here...
# report.add_mismatch(...) for each discrepancy

report.generate_report("reports/validation.html")

Report Features:

• 📊 KPI dashboard with pass/fail rates
• 🔍 Column-level filtering and search
• 📋 Row-by-row mismatch details
• 🎨 Color-coded status indicators
• 📱 Mobile-responsive design
• ⚡ Zero manual mismatch tracking required!

Examples:

• Simple API: demo/functional/database/simple_comparison_examples.py ⭐ Recommended
• Manual Building: demo/functional/database/data_comparison_with_html_report.py

Full Documentation

• docs/PIP_INSTALL_GUIDE.md - Complete patterns guide
• demo/README.md - Examples documentation
• docs/DATA_VALIDATION_GUIDE.md - Advanced validation

Unit Testing and Coverage

# Windows
scripts\run_unit_tests_with_coverage.bat

# Linux/macOS
bash scripts/run_unit_tests_with_coverage.sh

# View coverage report
# Open reports/coverage-html/index.html in browser

Coverage target: 95% (configured in pytest.ini)

Security Scanning

# Windows
scripts\run_security_reports.bat

# Linux/macOS
bash scripts/run_security_reports.sh

# View reports
# Open reports/security_reports/html/security_consolidated_report.html

Building Internal Package

# Windows
scripts\build_internal_package.bat

# Linux/macOS
bash scripts/build_internal_package.sh

Configuration

Framework Configuration

• config/config.yaml - Framework defaults
• config/environments.yaml - Global environment settings

Project Configuration

• config/projects/project_1.yaml - project_1 settings
• config/projects/project_2.yaml - project_2 settings
• config/projects/project_3.yaml - Customs settings

Each project config includes:

• Environment-specific URLs
• API endpoints
• Test users
• Feature flags

Documentation for QE Engineers

Quick Start

• QUICK_START_QE.md - Get started in 10 minutes
• TEST_LOCATION_CHEATSHEET.md - One-page reference
• COMMANDS_FOR_YOUR_PROJECTS.md - Command reference
• docs/PIP_INSTALL_GUIDE.md - Pip install guide with data comparison patterns

Implementation

• YOUR_PROJECTS_IMPLEMENTATION_GUIDE.md - Step-by-step guide
• CONTRIBUTING.md - Comprehensive contributor guide
• FINAL_PROJECT_SETUP.md - Complete setup overview

Architecture

• docs/MULTI_PROJECT_STRUCTURE.md - Multi-project architecture
• docs/TEST_ORGANIZATION.md - Organization strategy
• docs/QE_VISUAL_GUIDE.md - Visual guide with diagrams

Data Comparison & Validation

• demo/functional/database/data_comparison_examples.py - Runnable examples for Excel, CSV, DB comparison
• demo/README.md - Examples documentation
• docs/DATA_VALIDATION_GUIDE.md - Advanced validation patterns

Project Guides

• tests/projects/project_1/README.md - project_1 guide
• tests/projects/project_2/README.md - project_2 guide
• tests/projects/project_3/README.md - Customs guide

Key Features

Multi-Project Support

• Separate test suites per project
• Independent configurations
• Project-specific page objects and test data
• No merge conflicts between projects

Scalability

• Designed for 50-100 QE engineers
• Clear ownership boundaries
• Parallel development across projects
• Independent CI/CD pipelines

Test Types

• UI Testing: Playwright-based with page objects
• API Testing: REST API validation with response validators
• Database Testing: Data integrity and migration validation
• Integration Testing: Cross-layer consistency validation

Observability

• Session-level metrics and KPIs
• Automatic failure classification
• Flaky test detection
• Multi-persona reports (engineering, QA, leadership)

Resilience

• Self-healing locators with fallback chains
• DOM retry mechanisms
• Confidence scoring
• Optional audit logging

Plugin Architecture

• Extensible hook system
• Built-in plugins: API, Data, Security
• Config-gated activation
• Session and test-level hooks

Wave Implementation Status

Wave 1 ✅ Complete:

• Observability tracker
• Environment readiness checks
• Session-level reporting

Wave 2 ✅ Complete:

• Test data management utilities
• Multi-level persona reports
• Integration adapters (file and API)

Wave 3 ✅ Complete:

• Plugin architecture with registry
• Orchestration planner with dependency graphs

Wave 4 ✅ Complete:

• Self-healing locators with fallback chains
• Distribution packaging baseline
• Multi-project organization structure

Wave 5 ✅ Complete:

• MCP server for AI interoperability (11 read-only tools, 3 categories)
• CLI layer — every MCP tool as a pyplaykit-mcp <subcommand> shell command
• CLI version management with PyPI upgrade detection
• Multi-project init as default layout; upgrade-to-multi migration command
• Version resolution from installed package metadata
• Allure environment capture fix for Windows/Python 3.14

Wave 5+ ✅ Complete:

• Renamed examples/ → demo/ with layered functional/{ui,api,database}/ structure
• HTML comparison report path logged to console after every comparison run
• Dockerized SRC→TGT PostgreSQL comparison demo (demo/functional/database/postgres_src_tgt/)
• CSV / XLS / XLSX file-comparison demo with multi-sheet behavior documentation (demo/functional/database/file_comparison/)

Reports and Artifacts

Test Reports

• HTML report: reports/report.html
• JUnit XML: reports/junit.xml
• Coverage: reports/coverage-html/index.html

Observability

• Summary: reports/observability/summary.json
• KPI summary: reports/observability/kpi_summary.json
• Engineering report: reports/observability/engineering_report.json
• QA report: reports/observability/qa_functional_report.json
• Leadership report: reports/observability/leadership_kpi_report.json

Viewing observability as an HTML report: the reports above are JSON. To turn them into a single, self-contained, styled HTML report with a generation date/time stamp, run the observability-report CLI command after a pytest run:

# Default: reads reports/observability/, writes a timestamped HTML file there
pyplaykit observability-report

# Point at a different reports directory
pyplaykit observability-report --reports-dir path/to/observability

# Choose an explicit output file
pyplaykit observability-report --output reports/observability/latest.html

# Generate and open in the default web browser
pyplaykit observability-report --open

The report includes a brand header with the generation timestamp and an overall PASSED / ATTENTION-NEEDED status pill, KPI cards (total, passed, failed, skipped, pass rate, flaky count, durations, automation coverage), leadership KPI metrics, a failed-tests table with failure classification, and a flaky-tests list. It requires that a pytest run has already written summary.json. See docs/CLI_REFERENCE.md for full option details.

Integration Exports

• Jira export: reports/integrations/jira_export.json
• Test management: reports/integrations/test_management_export.json

Resilience

• Audit log: reports/resilience/recovery_audit.jsonl

Failure Artifacts

• Screenshots: reports/screenshots/
• Videos: reports/videos/
• Logs: reports/framework.log

Environment Variables

Framework

export PYPLAYKIT_TEST_USERNAME="standard_user"
export PYPLAYKIT_TEST_PASSWORD="secret_sauce"

project_1

export project_1_USERNAME="user"
export project_1_PASSWORD="pass"
export project_1_ADMIN_USERNAME="admin"
export project_1_ADMIN_PASSWORD="admin_pass"

project_2

export project_2_DISPATCHER_USERNAME="dispatcher"
export project_2_DISPATCHER_PASSWORD="pass"
export project_2_DRIVER_USERNAME="driver"
export project_2_DRIVER_PASSWORD="pass"
export project_2_ADMIN_USERNAME="admin"
export project_2_ADMIN_PASSWORD="admin_pass"

Customs Modernization

export CUSTOMS_USERNAME="customs_user"
export CUSTOMS_PASSWORD="pass"
export CUSTOMS_ADMIN_USERNAME="admin"
export CUSTOMS_ADMIN_PASSWORD="admin_pass"
export CUSTOMS_OFFICER_USERNAME="officer"
export CUSTOMS_OFFICER_PASSWORD="officer_pass"

CI/CD Integration

GitHub Actions Example

name: Multi-Project Tests

on: [push, pull_request]

jobs:
  project_1-smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          playwright install
      - name: Run project_1 Smoke Tests
        run: pytest tests/projects/project_1/ -m smoke -n 4

  project_2-smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          playwright install
      - name: Run project_2 Smoke Tests
        run: pytest tests/projects/project_2/ -m smoke -n 4

  customs-smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          playwright install
      - name: Run Customs Smoke Tests
        run: pytest tests/projects/project_3/ -m smoke -n 4

Team Organization

Recommended Structure (60 QE Engineers)

project_1 Team (20 QE)

• UI Team (10): tests/projects/project_1/functional/ui/
• API Team (6): tests/projects/project_1/functional/api/
• Integration Team (4): tests/projects/project_1/integration/

project_2 Team (20 QE)

• UI Team (10): tests/projects/project_2/functional/ui/
• API Team (6): tests/projects/project_2/functional/api/
• Integration Team (4): tests/projects/project_2/integration/

Customs Modernization Team (20 QE)

• UI Team (8): tests/projects/project_3/functional/ui/
• API Team (5): tests/projects/project_3/functional/api/
• Database/Migration Team (5): tests/projects/project_3/functional/database/
• Integration Team (2): tests/projects/project_3/integration/

Contributing

See CONTRIBUTING.md for comprehensive guidelines on:

• Test development patterns
• Page object creation
• Test data management
• Code quality standards
• Common patterns and examples

Troubleshooting

Tests not discovered?

pytest tests/projects/<project>/ --collect-only

Import errors?

# Activate virtual environment
.\.venv\Scripts\Activate.ps1  # Windows
source .venv/bin/activate      # Linux/Mac

Configuration issues?

pytest --markers | grep -E "project_1|project_2|customs"

Support

• Framework Issues: See Framework Documentation
• Project Setup: See Implementation Guide
• Quick Help: See Cheatsheet

License

[Your License Here]

Authors

• Framework Team
• QA Engineering Teams

PyPlayKit - Enterprise Test Automation at Scale 🚀