pytest-coveragemarkers 3.3.0

Using pytest markers to track functional coverage and filtering of tests

Using pytest markers to track functional coverage and filtering of tests

---

This pytest plugin was generated with Cookiecutter along with @hackebrot’s cookiecutter-pytest-plugin template.

Features

• Definition of CoverageMarkers© in YAML format

• Support for applying CoverageMarkers© to tests

• Filtering of tests based on CoverageMarkers©

• Inclusion of CoverageMarkers© in JSON report

Installation

You can install “pytest-coveragemarkers” from PyPI:

$ pip install pytest-coveragemarkers
# or
$ poetry add pytest-coveragemarkers

Usage

Step 1: Define your coverage markers yaml.

Using the format:

markers:
  - name: <marker_name>
    allowed:
      - <marker_value_1>
      - <marker_value_2>
  - name: <marker2_name>
    allowed:
      - <marker2_value_1>
      - <marker2_value_2>

Then decorate your tests with them

import pytest

@pytest.mark.marker_name(['value1', 'value2'])
@pytest.mark.marker2_name(['value1', 'value2'])
def test_x():
    ...

@pytest.mark.marker2_name(['value1', 'value2'])
def test_y():
    ...

Then when the tests are executed with

pytest --json-report --markers-location=/full/path/to/coverage_markers.yml

Then the JSON Test Report output from the test execution contains:

"tests": [
{
  "nodeid": "...",
  "metadata": {
    "cov_markers": {
      "marker_name": {
        "value1": true,
        "value2": true
      },
      "marker2_name": {
        "value1": true,
        "value2": true
      }
    }
  }
},
...
]

This can then be used to generate test coverage details based on the coverage markers. A nice demo will be produced to give examples of usage.

But wait there is another benefit:

We can filter tests for execution based on their coverage markers

pytest \
    --filter='"value1" in marker_name' \
    --json-report \
    --markers-location=/full/path/to/coverage_markers.yml

The above command run against the tests defined above would select ‘test_x’ and deselect ‘test_y’ for execution

Other examples of filters are:

You can also supply the path to a file containing your filter. Use argument –filter-location or key FilterLocation in the pytest.ini file.

Mandatory Coverage Markers

Coverage markers can be detailed as mandatory by including the mandatory attribute.

E.g.

markers:
  - name: <marker_name>
    mandatory: True
    allowed:
      - <marker_value_1>
      - <marker_value_2>

Dependent Coverage Markers

Coverage markers can be detailed as a dependency on another marker. This ensures that if a marker is specified all dependencies of this marker in the chain must also be specified.

E.g.

markers:
  - name: <marker_name>
    dependents:
      - <marker_name...>
      - <marker_name...>
    allowed:
      - <marker_value_1>
      - <marker_value_2>

Coverage Marker Argument Format

The arguments supplied to Coverage Markers can follow multiple formats which allows the user to define the format that best suites them.

E.g.

import pytest

@pytest.mark.marker_1('value1')                 # single string argument
@pytest.mark.marker_2('value1', 'value2')       # multiple string arguments
@pytest.mark.marker_3(['value1', 'value2'])     # list of arguments
@pytest.mark.marker_4(('value1', 'value2'))     # tuple of arguments
def test_x():
    ...

Testing

Nox is used by this project to execute all tests. To run a specific set of tests execute the below line:

$ poetry run nox -s <session_name>

Where session_name can be one of the following

Nox Sessions

Session Name	Session Details
unit_tests	Execute all tests marked as unit
functional_tests	Execute all tests marked as functional

Thought Process

• The pytest_docs talks about using markers to set metadata on tests and use the markers to select required tests for execution.

• For the markers I want to add, I also want to specify a list of values that go along with that marker. E.g. If the marker was ‘colour’ then supported values may be ‘Red’, ‘Green’, ‘Gold’.

• I also want the list of values validated against supported values so no unsupported values can be added. E.g. If the marker was ‘colour’ then a value of ‘Panda’ would not be allowed.

• Then all this meta data I want to come out in the junit json report.

• Next I want to use these markers and their supported values to filter tests. For this I need a more powerful filter engine.

Documentation

To build the docs run:

poetry run mkdocs serve

License

Distributed under the terms of the MIT license, “pytest-coveragemarkers” is free and open source software

Issues

If you encounter any problems, please file an issue along with a detailed description.

Future Changes

• Type-Hints

• Full Test Coverage

• Full Documentation