playwright-flake-triage 0.1.4

Local read-only triage for flaky Playwright test reports and logs.

Playwright Flake Triage Toolkit

Local, read-only CLI for turning Playwright reports and CI logs into a short triage report: suspected cause, confidence, and practical next fixes.

The tool is intentionally simple: it runs locally, has no telemetry, requires no account, and does not mutate project files.

What it scans

• Playwright JSON reporter output (.json)
• JUnit XML reports (.xml)
• CI/test logs (.log, .txt, .out, .err)
• Files or whole directories

What it detects

• Timeout / wait condition instability
• Ambiguous or brittle selectors, including strict-mode violations
• Auth/session or expected page state mismatch
• Network/backend dependency flakes
• Browser/context/page lifecycle races
• Visual/snapshot instability
• Parallel/shared-state collisions
• Unclassified failures for manual grouping

Quick start

Install the CLI and run it against local Playwright reports or CI logs:

python3 -m pip install playwright-flake-triage
pw-flake-triage path/to/playwright-report.json path/to/ci.log --format markdown

Write JSON output for automation:

pw-flake-triage path/to/reports-or-logs --format json -o triage-report.json

Add project-specific heuristics with a JSON rules config:

pw-flake-triage ci.log --rules-config examples/custom-rules.json

Use gate mode when you want CI to fail after the report is written:

pw-flake-triage test-results ci.log --fail-on-severity high
pw-flake-triage test-results ci.log --fail-on-findings

Append a Markdown report to a GitHub Actions job summary while keeping JSON for artifacts or follow-up steps:

pw-flake-triage test-results ci.log \
  --format json \
  --output triage-report.json \
  --github-step-summary

The CLI uses only local files you pass to it. It does not contact Playwright, GitHub, CI providers, or any external service.

Clone / development install

git clone https://github.com/vasiliy0/playwright-flake-triage.git
cd playwright-flake-triage
python3 -m pip install -e .
pw-flake-triage examples --format markdown

Custom rules

Use --rules-config to add local project/team heuristics without changing the built-in rule set. The option can be repeated.

{
  "rules": [
    {
      "id": "app-hydration",
      "label": "Application hydration race",
      "severity": "medium",
      "patterns": ["hydration failed", "client boot not complete"],
      "why": "The application may not have finished client-side hydration before the test interacted with it.",
      "fixes": ["Wait for a durable hydrated UI marker before interacting."],
      "priority": 80,
      "suppresses": ["timeout-wait"]
    }
  ]
}

Rules are local JSON files. They are not uploaded or sent anywhere.

CI gate modes

By default, the CLI is non-gating and exits 0 after writing the report, even when findings are detected. Add a gate flag when the triage result should fail a CI step:

• --fail-on-findings exits 1 if any finding is detected.
• --fail-on-severity high|medium|low exits 1 if any finding has severity at or above the threshold.

Reports are written before the non-zero exit, so CI jobs can still upload JSON artifacts or render $GITHUB_STEP_SUMMARY.

GitHub Actions summary

In GitHub Actions, --github-step-summary appends Markdown to the file path in $GITHUB_STEP_SUMMARY. The flag is safe to combine with --format json --output triage-report.json, so the workflow can show a readable summary and keep machine-readable output as an artifact.

The command fails with exit code 2 if the flag is used outside GitHub Actions without GITHUB_STEP_SUMMARY set.

Example output

# Playwright Flake Triage Report

Scanned files: **3**
Findings: **3**

## Summary by suspected cause
- **Auth/session or expected page state mismatch**: 2
- **Ambiguous or brittle selector**: 1

## Repeated failure groups
- **Ambiguous or brittle selector**: 2 findings
  - Tests: retry-1.log, retry-2.log

## Findings
### 1. Auth/session or expected page state mismatch
- Issue: Synthetic auth/session timeout symptom
- Source: https://github.com/example/repo/issues/123
- Suggested fixes:
  - Verify the test starts with the intended storage state, user role, and seeded data.

Optional issue metadata for validation snippets

Plain text logs can include lightweight source metadata:

Issue: Synthetic auth/session timeout symptom
URL: https://github.com/example/repo/issues/123

Playwright TimeoutError: Timeout 30000ms exceeded.
logs: waiting for get_by_label("Keep me signed in")

The metadata is echoed in Markdown/JSON reports, which makes issue-derived regression fixtures easier to review without copying long third-party issue text.

Why this is useful

Flaky test cleanup usually starts with a pile of traces and CI logs. This v1 does not claim to prove root cause; it groups common failure smells so a developer can decide where to look first.

Safety / privacy

• Runs locally only; no network calls or telemetry.
• Does not call external services.
• Does not mutate test reports or project files.
• Be careful before sharing generated reports: logs may contain internal URLs, user data, or secrets.

Release status

• License: MIT.
• Distribution: GitHub source release and TestPyPI dry-run package.
• Current release notes: docs/PUBLIC_RELEASE_NOTES_v0.1.0.md.

Docs

• docs/GITHUB_ACTIONS.md — CI usage with job summaries, JSON artifacts, and custom rules.
• docs/PUBLIC_RELEASE_NOTES_v0.1.0.md — release notes.
• SECURITY.md — privacy/sensitive-log notes.
• CONTRIBUTING.md — contribution boundaries.

Roadmap

• Parse Playwright trace zip metadata when available.
• Add configurable custom rules.
• Group duplicate failures across retries/workers.
• Emit GitHub Actions step summary markdown.
• Add sample remediation PR checklist.