test-reporting 3.3.0

Multi-project test reporting dashboard — collect results locally or push to S3

test-reporting

Multi-project test reporting dashboard. Collects pytest results, publishes to a central store (local folder or S3), and serves a clean dashboard anyone can read.

---

How it works

Project A: pytest → local SQLite → publish → ┐
Project B: pytest → local SQLite → publish → ┼─► Central store (local/S3) ─► Dashboard
Project C: pytest → local SQLite → publish → ┘

Local (per project):

• Tests run and results are saved to a local SQLite database (test_results.db)
• SQLite is a scratchpad for local commands (stats, suites, cleanup)
• SQLite is never pushed to the store

Central store (shared):

• test-report publish reads from SQLite and publishes JSON files to the store
• Store contains: run files, artifacts (screenshots/traces), and data.json
• Dashboard reads from the store and aggregates all projects

The HTML files are static — deployed once. The data.json is rebuilt after every publish.

---

Installation

pip install test-reporting

# With S3 support
pip install test-reporting[s3]

---

Quick start

1. Add reporting.ini to your project

[reporting]
project_name = my-project

store_type = local
store_path = /shared/test-reports

# auto_publish = true   # push results automatically after every test run

2. Deploy the dashboard (once, from any project)

test-report deploy

This copies the static HTML files to store_path. Only needs to be done once.

3. Run tests and publish

pytest
test-report publish

Or set auto_publish = true in reporting.ini to skip the manual publish step.

4. Open the dashboard

# Local
open /shared/test-reports/index.html

# S3 — your bucket's static website URL

---

Multi-project setup

Every project has its own reporting.ini with a unique project_name, all pointing to the same store_path or S3 bucket.

Project A:  project_name = frontend-tests,  store_path = /shared/reports
Project B:  project_name = api-tests,       store_path = /shared/reports
Project C:  project_name = e2e-tests,       store_path = /shared/reports

The dashboard aggregates all three automatically.

---

Test suite structure

The plugin detects the suite name from the pytest argument automatically:

pytest tests/abc/test_suite_abc.py   # suite_name = test_suite_abc
pytest tests/abc/test_file_a.py      # suite_name = test_file_a

Suites run on the same day are grouped as a regression in the dashboard.

Tagging regressions explicitly

Set REGRESSION_TAG as an environment variable when kicking off your suites. All runs with the same tag are grouped together:

# Jenkins — set in each job's environment
REGRESSION_TAG=sprint-42 pytest tests/abc/test_suite_abc.py

# Local
export REGRESSION_TAG=sprint-42
pytest tests/abc/test_suite_abc.py
pytest tests/def/test_suite_def.py

Or set it in reporting.ini:

regression_tag = sprint-42

---

Jenkins setup

Local store (shared network drive)

[reporting]
project_name = my-project
store_type = local
store_path = //network-share/test-reports

stage('Test') {
    sh 'pytest tests/'
}
stage('Publish') {
    sh 'test-report publish'
}

S3

[reporting]
project_name = my-project
store_type = s3
s3_bucket = my-test-reports-bucket
s3_prefix = reports/
s3_region = us-east-1

AWS credentials are read from environment variables or an IAM role — not stored in reporting.ini.

environment {
    REGRESSION_TAG = "2026-05-29"
    AWS_ACCESS_KEY_ID     = credentials('aws-access-key')
    AWS_SECRET_ACCESS_KEY = credentials('aws-secret-key')
}

stage('Test') {
    sh 'pytest tests/'
}
stage('Publish') {
    sh 'test-report publish'
}

---

Configuration reference

[reporting]
# Required
project_name = my-project

# Store
store_type = local          # local | s3
store_path = /shared/reports  # for store_type = local
# s3_bucket = my-bucket     # for store_type = s3
# s3_prefix = reports/
# s3_region = us-east-1

# Auto-publish after every test run (default: false)
auto_publish = false

# Regression grouping tag (can also be set via REGRESSION_TAG env var)
# regression_tag = sprint-42

# Local SQLite — used by stats/suites/cleanup commands
db_path = test_results.db
retention_days = 365

# Dashboard data retention (affects data.json size and loading speed)
dashboard_max_days = 90           # Days of history in dashboard (default: 90)
dashboard_max_runs_per_suite = 50 # Run history per suite (default: 50)

Note: retention_days controls manual cleanup commands, while dashboard_max_days and dashboard_max_runs_per_suite control what appears in the dashboard automatically. Individual run files remain accessible regardless of these settings.

---

CLI commands

Command	Description
test-report publish	Push latest test run to the store
test-report deploy	Deploy static dashboard HTML to the store (run once)
test-report open	Open local dashboard in browser
test-report stats	Show stats from latest run (reads local SQLite)
test-report suites	Show suite statistics (reads local SQLite)
test-report cleanup	Delete old runs from local SQLite
test-report cleanup-artifacts	Delete old screenshots and traces (preserves data)

Cleaning up old data

Both cleanup commands use the retention_days setting from reporting.ini (default: 365 days).

test-report cleanup

Removes all test data from the local SQLite database only:

• Deletes test runs, results, steps, logs, and analytics older than retention_days
• Use this to free up local database space
• Does not affect the published store (JSON files, artifacts, or dashboard data remain intact)
• Only impacts local commands like stats and suites

test-report cleanup-artifacts

Removes only screenshots and traces while preserving all test data:

test-report cleanup-artifacts

This will:

• Remove screenshots (.png) and traces (.zip) older than retention_days
• Clean local directories (screenshots/, traces/)
• Clean published store artifacts (store_path/artifacts/ for local stores)
• Show file count and total size before deletion
• Ask for confirmation
• Preserve all local SQLite data and published JSON files (logs, error messages, analytics remain intact)

Why use this? Traces can be 10-50 MB each. This frees disk space on both local machines and the published store while keeping all test data for trend analysis.

What happens in the UI? The dashboard gracefully handles missing artifacts:

• Screenshots show: "Screenshot cleaned up" (dashed border placeholder)
• Trace links show: "Trace file cleaned up" (yellow badge)
• All test data, logs, and error messages remain visible

Note: Only failed tests have screenshots and traces saved. Passed tests don't generate these artifacts.

S3 limitation: Currently only cleans local artifacts. S3 artifact cleanup will be added in a future version.

Configure retention in reporting.ini:

retention_days = 90  # Keep 90 days of artifacts and data

---

Dashboard pages

index.html — All projects at a glance. Pass rate, health status, last run time.

project.html?p=<name> — Single project view. Regression history grouped by tag or date, suite run history, trend chart.

run.html?r=<id>&p=<name> — Run detail. Test file breakdown, every failing test with its error message, screenshot and trace links.

---

Migrating from v1

1. pip install --upgrade test-reporting
2. Add store_type, store_path (or S3 config) to reporting.ini
3. Run test-report deploy once
4. Replace test-report generate with test-report publish in your pipeline

Old commands that still work: stats, suites, cleanup

---

Project structure

reporting/
  plugin.py       — pytest plugin (auto-collects results)
  publisher.py    — writes run files + rebuilds data.json
  config.py       — reads reporting.ini / env vars
  classifier.py   — categorises failures
  storage.py      — local SQLite
  cli.py          — CLI entry point
  templates/
    index.html    — projects landing page
    project.html  — per-project view
    run.html      — run detail view

---

Requirements

• Python >= 3.9
• pytest >= 8.0.0
• boto3 >= 1.26.0 (only for S3 — pip install test-reporting[s3])

---

License

MIT