pytest-vigil 0.7.0

A pytest plugin for enhanced test reliability and monitoring

Pytest Vigil

Pytest Vigil is a reliability pytest plugin that enforces resource limits on your tests and kills them when they exceed those limits.

Why you might need this

• Tests sometimes hang indefinitely due to deadlocks or infinite loops
• Memory leaks crash your test runner or CI environment
• CPU-intensive tests slow down your entire suite
• You want to enforce maximum runtime for your CI pipeline
• You need to identify which tests are resource hogs

---

✨ Features

• Deadlock detection: Kill tests that hang with low CPU activity
• Suite timeout: Stop the entire test run after a specified duration
• CI scaling: Automatically relaxes limits by 2x in CI environments (configurable)
• Retry mechanism: Re-run tests that fail due to resource violations
• Detailed reports: JSON output showing CPU breakdown by process type (browser, renderer, etc.)
• Per-test timeout: Layered timeout enforcement — kernel-level alarm backstop, C-level faulthandler diagnostics, and optional force-exit escalation for tests stuck in GIL-holding C extensions
• Per-test memory limit: Kills tests that exceed memory limits
• Per-test CPU limit: Kills tests that exceed CPU usage limits

🚀 Installation

uv add -D pytest-vigil
# or
pip install pytest-vigil

⚡ Quick Start

1. Protect against heavy tests Limit tests to 5 seconds, 512MB RAM, and 80% CPU:

pytest --vigil-timeout 5 --vigil-memory 512 --vigil-cpu 80

2. Prevent infinite CI hangs Kill the entire suite if it runs longer than 15 minutes:

pytest --vigil-session-timeout 900

3. Generate Reliability Report

pytest --vigil-json-report
# saved to .pytest_vigil/vigil_report.json

🛠 Usage & Configuration

CLI Options Reference

Option	Default	Description
--vigil-timeout	None	Max duration per test (seconds)
--vigil-memory	None	Max memory usage (MB)
--vigil-cpu	None	Max CPU usage (%)
--vigil-retry	0	Auto-retry failed/limit-violating tests
--vigil-stall-timeout	None	Max duration of low CPU (deadlock detection)
--vigil-session-timeout	None	Global timeout for entire test run
--vigil-json-report	False	Enable JSON reliability report saving
--vigil-output-dir	.pytest_vigil	Base directory for generated Vigil artifacts
--vigil-cli-report-verbosity	short	Terminal output: none, short, full
--vigil-force-exit-delay	None	Seconds after a soft interrupt before calling os._exit(124). Use when tests are stuck in GIL-holding C extensions. Disabled by default.

Using Markers

Apply specific limits to critical or heavy tests directly in code. All arguments are optional.

Parameter	Type	Unit	Default	Description
timeout	float	s	None	Test timeout
memory	float	MB	None	Memory limit
cpu	float	%	None	CPU limit
retry	int	-	0	Number of retries on failure
stall_timeout	float	s	None	Max duration of low CPU activity
stall_cpu_threshold	float	%	1.0	CPU threshold for stall detection

import pytest

@pytest.mark.vigil(timeout=5.0, memory=512, retry=2)
def test_heavy_computation():
    ...

Environment Variables

Perfect for CI/CD pipelines. All options are available via PYTEST_VIGIL__* prefix.

Variable	Default	Description
TIMEOUT	None	Default test timeout (seconds)
MEMORY_LIMIT_MB	None	Default memory limit (MB)
CPU_LIMIT_PERCENT	None	Default CPU limit (%)
SESSION_TIMEOUT	None	Global suite timeout (seconds)
SESSION_TIMEOUT_GRACE_PERIOD	5.0	Seconds to wait for graceful shutdown
MONITOR_INTERVAL	0.1	Internal check frequency (seconds)
STRICT_MODE	True	Enforce strict monitoring
CI_MULTIPLIER	2.0	Limit multiplier for CI environments
RETRY_COUNT	0	Number of retries for failures
STALL_TIMEOUT	None	Low-CPU deadlock timeout (seconds)
STALL_CPU_THRESHOLD	1.0	Threshold (%) for stall detection
CONSOLE_REPORT_VERBOSITY	short	Terminal output: none, short, full
JSON_REPORT_FILENAME	vigil_report.json	Default JSON report filename when reporting is enabled
JSON_REPORT	False	Enable JSON report saving by default
ARTIFACTS_DIR	.pytest_vigil	Base directory for generated Vigil artifacts
FORCE_EXIT_DELAY	None	Seconds after a soft interrupt before calling os._exit(124). Disabled by default.

📊 Reporting

Vigil provides insights into where your resources are going.

Terminal Report

Control verbosity with --vigil-cli-report-verbosity (none, short, full).

Short Mode (Default):

Vigil Reliability Report
Total Tests: 953 | Avg Duration: 5.32s | Avg Memory: 288.6 MB

Peak CPU by Process Type:
  Browser: 3542.1%  (Chromium/Webkit)
  Renderer: 2156.8% (Tab rendering)
  Pytest: 593.5%    (Test logic)

Full Mode:

Vigil Reliability Report
Test ID                                                 Att Duration (s)  Max CPU (%) Max Mem (MB)
--------------------------------------------------------------------------------------------------
tests/test_stress.py::test_high_load                      0         8.42        450.5        820.1
tests/test_ui.py::test_login[chromium]                    0         4.15       2101.2        415.8
tests/test_ui.py::test_checkout[chromium]                 1        12.30       3542.1        590.4
tests/test_api.py::test_latency                           0         0.25         15.2         45.1

💡 Note on CPU > 100%: In multi-process testing (like Playwright/Selenium), usage is summed across all cores and child processes. 7000% CPU usage means your test suite is utilizing ~70 cores efficiently (or inefficiently!).

JSON Report

The JSON report captures cpu_breakdown for every test, helping you identify if it's the Browser, DB, or Python code causing the spike.

Report saving is disabled by default. Enable it with --vigil-json-report; output goes to .pytest_vigil/vigil_report.json unless overridden.

Key Fields:

• flaky_tests: Tests that passed after retry (attempt > 0)
• cpu_breakdown: Peak CPU by process type (pytest, browser, renderer, gpu, webdriver, python, automation)
• limits: Applied resource constraints from CLI/markers/env

📄 Example JSON Report (click to expand)

{
  "timestamp": "2026-02-08T14:23:45.123456+00:00",
  "flaky_tests": [
    "tests/test_integration.py::test_api_retry"
  ],
  "results": [
    {
      "node_id": "tests/test_ui.py::test_checkout[chromium]",
      "attempt": 0,
      "duration": 12.34,
      "max_cpu": 3542.1,
      "max_memory": 590.4,
      "cpu_breakdown": {
        "pytest": 89.2,
        "browser": 1805.3,
        "renderer": 1247.6,
        "gpu": 400.0
      },
      "limits": [
        {
          "limit_type": "time",
          "threshold": 15.0,
          "secondary_threshold": null,
          "strict": true
        },
        {
          "limit_type": "memory",
          "threshold": 1024.0,
          "secondary_threshold": null,
          "strict": true
        }
      ]
    },
    {
      "node_id": "tests/test_integration.py::test_api_retry",
      "attempt": 1,
      "duration": 2.15,
      "max_cpu": 45.8,
      "max_memory": 128.3,
      "cpu_breakdown": {
        "pytest": 45.8
      },
      "limits": [
        {
          "limit_type": "time",
          "threshold": 5.0,
          "secondary_threshold": null,
          "strict": true
        }
      ]
    }
  ]
}

⚖️ License

MIT