pytest-perfguard

pytest-perfguard records CPU and memory metrics for ordinary pytest tests and compares them with compatible historical results. It does not require benchmark fixtures, decorators, or markers.

This repository currently contains a proof of concept. Its goals are:

• measure the existing test run instead of repeating tests;
• use Linux hardware instruction counters when permitted;
• fall back explicitly to process CPU time when counters are unavailable;
• collect process RSS with no sampling thread or binary dependency;
• keep history in GitLab job artifacts rather than an external service;
• catch both single-test jumps and suite-wide accumulation;
• retain a fixed anchor so gradual drift remains visible after the rolling baseline adapts;
• keep performance regressions non-blocking without masking functional failures.

Install

Once the proof of concept is released, add the package to the same environment as pytest:

uv add --dev pytest-perfguard==0.1.0

The plugin is automatically discovered through pytest's pytest11 entry point. CI enables collection with:

PERFGUARD=1 pytest

It writes one artifact per GitLab test job under .perfguard/current/<CI_JOB_NAME_SLUG>-<CI_JOB_ID>.json, avoiding collisions when GitLab downloads several jobs' artifacts. With pytest-xdist, workers send their results to the controller and the controller writes one job-level file.

Non-blocking GitLab design

Do not put allow_failure: true on the normal test job: that would also make functional test failures non-blocking.

Instead, retain the normal blocking test job and add small report jobs. The default-branch job keeps the rolling database, while merge-request output expires:

.perfguard-report:
  stage: report
  needs:
    - job: test
      artifacts: true
  script:
    - uvx --from "pytest-perfguard==${PERFGUARD_VERSION}" pytest-perfguard gitlab
  when: always
  allow_failure: true
  artifacts:
    when: always
    paths:
      - .perfguard/results.json
      - .perfguard/junit.xml
    reports:
      junit: .perfguard/junit.xml

perfguard:
  extends: .perfguard-report
  artifacts:
    expire_in: never
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: always

perfguard-mr:
  extends: .perfguard-report
  artifacts:
    expire_in: 30 days
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: always

The complete example is in examples/gitlab-ci.yml. Projects with several test jobs list each producer under needs. On a merge request, the command downloads the latest successful target-branch perfguard artifact using CI_JOB_TOKEN. On the default branch it records the current samples into rolling history while preserving the fixed anchor. No GitLab credentials enter the application test container.

The result job returns:

• 0 when no confirmed regression is found, including warm-up and persistent drift;
• 1 when a regression is found;
• 2 when artifact preparation or comparison fails.

Because only this reporting job has allow_failure: true, all three outcomes remain visible without weakening the test gate.

The command prints the largest regressions and persistent drifts directly in the job log and writes a JUnit report. Confirmed regressions are failures; drift is a non-failing skipped testcase. Node IDs, metrics, baselines, and limits therefore appear in GitLab without downloading JSON.

Baseline policy

Results are compared only when architecture, CPU model, measurement backends, xdist worker count, and runner key match. On GitLab the runner key defaults to CI_RUNNER_DESCRIPTION, then CI_RUNNER_ID; PERFGUARD_RUNNER_KEY can group a known-homogeneous pool or separate another CI provider's machines. This prevents different shared-runner hosts from teaching one noisy baseline. Python version, lock hash, and commit SHA are recorded as diagnostic context rather than compatibility keys, so interpreter and dependency upgrades are compared.

Each compatible series keeps two baselines with different purposes:

• a rolling history estimates current noise with median and MAD;
• a fixed anchor is created automatically after three successful default-branch samples and never moves during normal recording.

A measurement that exceeds the rolling limit is a confirmed regression. If it exceeds only the fixed-anchor limit, it is reported as persistent drift without a failing exit code. Requiring rolling corroboration prevents a short, unusually fast anchor from making every later stable run flaky, while the fixed anchor still prevents gradual degradation from disappearing silently as history moves. To intentionally rebase after an understood change, run one successful default-branch pipeline with either:

pytest-perfguard gitlab --accept-baseline

or PERFGUARD_ACCEPT_BASELINE=1. Acceptance is rejected on merge-request and other compare-only pipelines. Do not leave the variable enabled permanently.

The initial thresholds require both a relative increase and an absolute floor:

Three compatible historical samples are required before gating. Thresholds and history length can be changed by CLI options.

Job aggregates are compared only when the set of passed test node IDs matches. They catch regressions that are individually below the absolute floor, such as 1,000 tests each becoming 2 ms slower or several tests each retaining 8 MiB. Individual baselines use the source of the test function, so editing a different test in the same file does not reset them.

Per-test process CPU remains in the raw artifact, but becomes a regression attribution only when the corresponding job aggregate also exceeds its rolling limit. Shared runners can move unrelated process work, garbage collection, and background activity between individual pytest calls; allowing an isolated call spike to fail the report produced false positives in real xdist trials. Per-test instruction counts and RSS growth remain independently guarded.

Measurement scope and limitations

• Linux instruction counting uses perf_event_open. Rootless or restricted runners commonly deny it; the artifact then says process_time, never instructions.
• CPU time is process CPU consumed during each pytest call. Hardware instructions count only the thread running that call. The artifact records these scopes explicitly. Summed job and session CPU remain guarded so threaded work is not silently ignored; individual CPU-time outliers are reported only as attribution for a corroborated job regression.
• RSS uses /proc/self/statm on Linux, working set on Windows, and peak RSS as a fallback. With xdist, process RSS is summed across the controller and workers, and per-test CPU is summed across workers; the artifact records the process count.
• RSS growth is a regression signal, not proof of a memory leak. Allocation tracing and repeated workloads remain diagnostic follow-ups.
• The proof of concept measures the pytest process. Separate application services require a service- or cgroup-level collector.
• Shared runners with different CPU models build separate history buckets.
• The default-branch .perfguard/results.json is the baseline database and uses expire_in: never; deleting it restarts warm-up. Merge-request comparison artifacts expire after 30 days in the example.