pytest-coveragemarkers 2.1.0

Using pytest markers to track functional coverage and filtering of tests

Using pytest markers to track functional coverage and filtering of tests

---

This pytest plugin was generated with Cookiecutter along with @hackebrot’s cookiecutter-pytest-plugin template.

Features

• Definition of CoverageMarkers© in YAML format

• Support for applying CoverageMarkers© to tests

• Filtering of tests based on CoverageMarkers©

• Inclusion of CoverageMarkers© in JSON report

Installation

You can install “pytest-coveragemarkers” from PyPI:

$ pip install pytest-coveragemarkers
# or
$ poetry add pytest-coveragemarkers

Usage

Step 1: Define your coverage markers yaml.

Using the format:

markers:
  - name: <marker_name>
    allowed:
      - <marker_value_1>
      - <marker_value_2>
  - name: <marker2_name>
    allowed:
      - <marker2_value_1>
      - <marker2_value_2>

Then decorate your tests with them

import pytest

@pytest.mark.marker_name(['value1', 'value2'])
@pytest.mark.marker2_name(['value1', 'value2'])
def test_x():
    ...

@pytest.mark.marker2_name(['value1', 'value2'])
def test_y():
    ...

Then when the tests are executed with

pytest --json-report --markers-location=/full/path/to/coverage_markers.yml

Then the JSON Test Report output from the test execution contains:

"tests": [
{
  "nodeid": "...",
  "metadata": {
    "cov_markers": {
      "marker_name": {
        "value1": true,
        "value2": true
      },
      "marker2_name": {
        "value1": true,
        "value2": true
      }
    }
  }
},
...
]

This can then be used to generate test coverage details based on the coverage markers. A nice demo will be produced to give examples of usage.

But wait there is another benefit:

We can filter tests for execution based on their coverage markers

pytest \
    --filter='{"and": [{"eq": ["marker_name.value1", true]}]}' \
    --json-report \
    --markers-location=/full/path/to/coverage_markers.yml

The above command run against the tests defined above would select ‘test_x’ and deselect ‘test_y’ for execution

Other examples of filters are:

You can also supply the path to a json file containing your filter. Use argument –filter-location or key FilterLocation in the pytest.ini file.

Coverage Marker Argument Format

The arguments supplied to Coverage Markers can follow multiple formats which allows the user to define the format that best suites them.

E.g.

import pytest

@pytest.mark.marker_1('value1')                 # single string argument
@pytest.mark.marker_2('value1', 'value2')       # multiple string arguments
@pytest.mark.marker_3(['value1', 'value2'])     # list of arguments
@pytest.mark.marker_4(('value1', 'value2'))     # tuple of arguments
def test_x():
    ...

Testing

Nox is used by this project to execute all tests. To run a specific set of tests execute the below line:

$ poetry run nox -s <session_name>

Where session_name can be one of the following

Nox Sessions

Session Name	Session Details
unit_tests	Execute all tests marked as unit
functional_tests	Execute all tests marked as functional

Thought Process

• The pytest_docs talks about using markers to set metadata on tests and use the markers to select required tests for execution.

• For the markers I want to add, I also want to specify a list of values that go along with that marker. E.g. If the marker was ‘colour’ then supported values may be ‘Red’, ‘Green’, ‘Gold’.

• I also want the list of values validated against supported values so no unsupported values can be added. E.g. If the marker was ‘colour’ then a value of ‘Panda’ would not be allowed.

• Then all this meta data I want to come out in the junit json report.

• Next I want to use these markers and their supported values to filter tests. For this I need a more powerful filter engine.

Documentation

To build the docs run:

poetry run mkdocs serve

License

Distributed under the terms of the MIT license, “pytest-coveragemarkers” is free and open source software

Issues

If you encounter any problems, please file an issue along with a detailed description.

Future Changes

• Type-Hints

• Full Test Coverage

• Full Documentation

• Refactor/rewrite

• Pick one, yml or json?