test-reporting 1.3.0

Beautiful multi-project test reporting dashboard system

🚀 Test Reporting System

A beautiful, multi-project test reporting dashboard system for Playwright/Pytest with shared database support.

---

✨ Features

• 📊 Beautiful Dashboards - Interactive HTML dashboards with charts and metrics
• 🎯 Multi-Project Support - Multiple projects share one database
• 🔍 Advanced Analytics - Flaky tests, failure patterns, performance tracking
• 📈 Trend Analysis - Track quality over time
• 🎨 Projects Landing Page - Overview of all your test projects
• 🔒 Concurrent Safe - SQLite WAL mode for parallel test runs
• 📦 Easy Integration - Simple pip install and config

---

🚀 Quick Start

Installation

# Install from PyPI
pip install test-reporting

# Or install from source for development
git clone https://github.com/yourusername/test-reporting.git
cd test-reporting
pip install -e .

Setup in Your Test Project

1. Add to conftest.py:

# conftest.py
pytest_plugins = ['reporting.plugin']

2. Create reporting.ini:

# reporting.ini
[reporting]
project_name = My Test Project
db_path = test_results.db
dashboard_dir = dashboard
retention_days = 365

3. Run tests:

pytest

4. Generate dashboards:

test-report generate
# or
python -m reporting.cli generate

5. Open dashboards:

test-report open

---

📊 Dashboards

Projects Landing Page

• Overview of all projects
• Health scores
• Quick metrics
• Click to drill down

Overview Dashboard

• Overall health metrics
• Module breakdown
• Failure hotspots
• Quality trends
• Flaky tests
• Performance metrics

Run Details Dashboard

• Latest test run details
• Test-by-test breakdown
• Screenshots and traces
• Error messages
• Execution logs

Test History

• All test runs
• Search and filter
• Trend analysis
• Compare runs

---

🎯 Multi-Project Setup

Shared Database Workflow

All projects can share one database. Here's how it works:

workspace/
├── shared_test_results.db          ← Single database
│
├── project-a/
│   ├── tests/
│   └── reporting.ini               ← project_name = Project A
│                                      db_path = ../shared_test_results.db
│
├── project-b/
│   ├── tests/
│   └── reporting.ini               ← project_name = Project B
│                                      db_path = ../shared_test_results.db
│
└── project-c/
    ├── tests/
    └── reporting.ini               ← project_name = Project C
                                       db_path = ../shared_test_results.db

How It Works:

1. Running Tests - Each project stores its test results in the shared database:

   cd project-a
   pytest  # Stores results tagged with "Project A"
   
   cd ../project-b
   pytest  # Stores results tagged with "Project B"
   

2. Generating Dashboards - Run from ANY project to generate dashboards with ALL data:

   cd project-a
   test-report generate  # Pulls ALL projects from shared DB
   

   The generate command:

   • ✅ Reads ALL project data from the shared database
   • ✅ Generates a projects landing page showing all projects
   • ✅ Creates overview, history, and run-details dashboards
   • ✅ Data from all projects is preserved and accessible

3. Viewing Results - The dashboards show aggregated data:

   • Projects Landing Page (index.html) - Lists all projects with health scores
   • Overview Dashboard (overview.html) - Shows metrics across all projects
   • History Dashboard (history.html) - All test runs from all projects
   • Run Details (run-details.html) - Latest run details

Key Benefits:

• 🔄 Data Persistence - Running test-report generate from one project doesn't lose data from other projects
• 📊 Centralized View - See all your test projects in one place
• 🎯 Project Filtering - Dashboards support filtering by project (future enhancement)
• 🔒 Concurrent Safe - Multiple projects can run tests simultaneously

---

⚙️ Configuration

reporting.ini

[reporting]
# Required: Unique project identifier
project_name = My Test Project

# Required: Database location (can be shared across projects)
db_path = test_results.db

# Optional: Dashboard output directory (default: dashboard)
dashboard_dir = dashboard

# Optional: Data retention in days (default: 365)
retention_days = 365

Environment Variables (Alternative)

export PROJECT_NAME="My Test Project"
export DB_PATH="test_results.db"

Config file takes precedence if it exists.

---

🛠️ CLI Commands

# Generate all dashboards
test-report generate

# Generate and open in browser
test-report open

# Show quick stats
test-report stats

# Show suite statistics
test-report suites

# Clean up old data
test-report cleanup

---

📈 Metrics Tracked

Top 10 Quick Win Metrics:

1. Retry Analysis - Tests that needed retries
2. Failure Type Breakdown - Categorized failures
3. Time to Recovery - How long failures persist
4. Performance Regressions - Slower tests over time
5. Failure Correlations - Tests that fail together
6. Error Message Search - Common error patterns
7. Test Age Analysis - Old vs new test stability
8. Soft Fail Analysis - Timeout and flaky patterns
9. Step-Level Failures - Where tests break
10. Module Health - Health by test file

Core Metrics:

• Pass rate (overall and functional)
• Test duration
• Flaky test detection
• Failure hotspots
• Quality trends
• Module health

---

🔒 Concurrency

Uses SQLite WAL mode for safe concurrent writes:

# Multiple projects can run tests simultaneously
Project A → test_results.db ← Safe!
Project B → test_results.db ← Safe!
Project C → test_results.db ← Safe!

---

🎨 Screenshots

Projects Landing Page

┌────────────────────────────────────────┐
│  🚀 Test Automation Projects           │
├────────────────────────────────────────┤
│  ┌──────────────────────────────────┐  │
│  │ ✅ Project A                     │  │
│  │ Health Score: 95%                │  │
│  │ 120 Tests | 50 Runs              │  │
│  │ [View Dashboard →]               │  │
│  └──────────────────────────────────┘  │
│  ┌──────────────────────────────────┐  │
│  │ ⚠️ Project B                     │  │
│  │ Health Score: 78%                │  │
│  │ 85 Tests | 30 Runs               │  │
│  │ [View Dashboard →]               │  │
│  └──────────────────────────────────┘  │
└────────────────────────────────────────┘

---

🔧 Development

Setup for Development

# Clone repo
git clone https://github.com/yourusername/test-reporting.git
cd test-reporting

# Install in editable mode
pip install -e .

# Make changes
# Test in your project

Project Structure

test-reporting/
├── reporting/
│   ├── __init__.py
│   ├── plugin.py              # Pytest plugin
│   ├── storage.py             # Database operations
│   ├── config.py              # Configuration
│   ├── cli.py                 # CLI commands
│   ├── classifier.py          # Failure classification
│   ├── overview_dashboard.py  # Overview dashboard
│   ├── dashboard_generator_v2.py  # Run details
│   ├── history_dashboard.py   # Test history
│   └── projects_dashboard.py  # Projects landing
├── setup.py
├── README.md
├── requirements.txt
└── .gitignore

---

📦 Requirements

• Python >= 3.9
• pytest >= 8.0.0
• playwright >= 1.40.0 (optional, for web tests)

---

🚀 CI/CD Integration

Jenkins

stage('Run Tests') {
    steps {
        sh 'pytest'
        sh 'test-report generate'
        archiveArtifacts artifacts: 'dashboard/**/*'
    }
}

GitHub Actions

- name: Run Tests
  run: pytest

- name: Generate Reports
  run: test-report generate

- name: Upload Dashboards
  uses: actions/upload-artifact@v2
  with:
    name: test-dashboards
    path: dashboard/

---

📝 License

MIT License - See LICENSE file

---

🤝 Contributing

1. Fork the repo
2. Create feature branch (git checkout -b feature/amazing-feature)
3. Make changes
4. Test thoroughly
5. Commit your changes (git commit -m 'Add amazing feature')
6. Push to branch (git push origin feature/amazing-feature)
7. Open a Pull Request

---

📧 Support

• Issues: https://github.com/yourusername/test-reporting/issues
• Documentation: See README and SETUP_GUIDE.md

---

🎉 Credits

Built with ❤️ for better test reporting

---

Happy Testing! 🚀📊✨