pytest-brightest

What is pytest-brightest? It is a pytest plugin that uses the brightest ideas in software testing research to illuminate problems with your test suite and brighten up the execution of your tests.

Table of Contents

• Perspectives
• Why pytest-brightest?
• Requirements
• Installation
  • Using uv
  • Using poetry
• Modes of Operation
  • Reordering
  • Shuffling
• Command-Line Arguments
  • Examples
• Contributing and Reporting Issues

Perspectives

pytest-brightest is a tool that helps software engineers run their test suites more effectively. Here are three perspectives on how it can be useful:

• The Busy Developer: "I need to get feedback from my tests as quickly as possible. By using pytest-brightest to run the fastest tests first, I can find out if I've introduced any simple bugs in seconds, rather than waiting minutes for the whole suite to run."
• The Quality Assurance Engineer: "My team is struggling with flaky tests that pass or fail unpredictably. pytest-brightest's shuffling capabilities help us shine a light on these tests by breaking hidden dependencies, making our test suite more reliable."
• The Open-Source Maintainer: "When a contributor submits a pull request, I want to be confident it doesn't re-introduce old bugs. Reordering tests to run previously failing ones first gives me a faster signal, streamlining my review process."

Why pytest-brightest?

In software testing, not all test executions are created equal. Some tests are long, others are short. Some are prone to failure, while others are steadfastly reliable. pytest-brightest helps you shine a light on these characteristics to run your test suite more intelligently. It's not just about leveraging the brightest ideas from software testing research; it's about making your tests brighter (yes, the pun is intended!). Whether you want to get feedback faster, find flaky tests, or simply bring more clarity to your testing chaos, pytest-brightest is here to help.

Requirements

To use pytest-brightest, you'll need:

• pytest
• pytest-json-report

pytest-brightest uses the data generated by pytest-json-report to work its magic. If you have pytest-json-report installed, pytest-brightest will automatically configure it for you.

Installation

Using uv

To install pytest-brightest using uv, a fast Python package installer and resolver, run the following command:

uv add pytest-brightest --group dev

Using poetry

To install pytest-brightest using poetry, a tool for dependency management and packaging in Python, run the following command:

poetry add pytest-brightest --group dev

Modes of Operation

pytest-brightest is enabled with the --brightest flag and can operate in several modes, configured through the --reorder-by-technique option.

Reordering

You can reorder your tests based on a variety of criteria:

• cost: Reorders tests based on their execution time from the previous run. This is perfect for getting quick feedback by running faster tests first.
• name: Reorders tests alphabetically by their name.
• failure: Reorders tests based on their failure history, running tests that failed in the previous run first. This is a powerful technique for regression testing.

Shuffling

• shuffle: Randomizes the order of your tests. This is an excellent way to uncover hidden dependencies between tests and identify "flaky" tests that pass or fail based on the order in which they are run.

Command-Line Arguments

Here's a breakdown of the command-line arguments you can use to control pytest-brightest:

• --brightest
  • Description: Enable the pytest-brightest plugin.
  • Default: False
• --reorder-by-technique
  • Description: The technique to use for reordering or shuffling tests.
  • Options: shuffle, name, cost, failure
  • Default: None
• --reorder-by-focus
  • Description: The scope of the reordering or shuffling.
  • Options: modules-within-suite, tests-within-module, tests-across-modules
  • Default: tests-across-modules
• --reorder-in-direction
  • Description: The direction of the reordering.
  • Options: ascending, descending
  • Default: ascending
• --seed
  • Description: A seed for the random number generator used for shuffling, ensuring reproducible results.
  • Default: (random)

Examples

Run faster tests first

pytest --brightest --reorder-by-technique cost --reorder-in-direction ascending

Run previously failing tests first

pytest --brightest --reorder-by-technique failure --reorder-in-direction descending

Shuffle all tests with a specific seed

pytest --brightest --reorder-by-technique shuffle --seed 12345

Shuffle tests within each module, but keep the module order

pytest --brightest --reorder-by-technique shuffle --reorder-by-focus tests-within-module

Contributing and Reporting Issues

Contributions are welcome! This project is open-source and thrives on community involvement. Let's make testing brighter, together.

• Issues: If you find a bug or have a feature request, please open an issue.
• Pull Requests: If you'd like to contribute code, please submit a pull request.