Using pytest markers to track functional coverage and filtering of tests
Project description
Using pytest markers to track functional coverage and filtering of tests
This pytest plugin was generated with Cookiecutter along with @hackebrot’s cookiecutter-pytest-plugin template.
Features
Definition of CoverageMarkers© in YAML format
Support for applying CoverageMarkers© to tests
Filtering of tests based on CoverageMarkers©
Inclusion of CoverageMarkers© in JSON report
Installation
You can install “pytest-coveragemarkers” from PyPI:
$ pip install pytest-coveragemarkers # or $ poetry add pytest-coveragemarkers
Usage
Step 1: Define your coverage markers yaml.
Using the format:
markers:
- name: <marker_name>
allowed:
- <marker_value_1>
- <marker_value_2>
- name: <marker2_name>
allowed:
- <marker2_value_1>
- <marker2_value_2>
Then decorate your tests with them
import pytest
@pytest.mark.marker_name(['value1', 'value2'])
@pytest.mark.marker2_name(['value1', 'value2'])
def test_x():
...
@pytest.mark.marker2_name(['value1', 'value2'])
def test_y():
...
Then when the tests are executed with
pytest --json-report --markers-location=/full/path/to/coverage_markers.yml
Then the JSON Test Report output from the test execution contains:
"tests": [
{
"nodeid": "...",
"metadata": {
"cov_markers": {
"marker_name": {
"value1": true,
"value2": true
},
"marker2_name": {
"value1": true,
"value2": true
}
}
}
},
...
]
This can then be used to generate test coverage details based on the coverage markers. A nice demo will be produced to give examples of usage.
But wait there is another benefit:
We can filter tests for execution based on their coverage markers
pytest \
--filter='"value1" in marker_name' \
--json-report \
--markers-location=/full/path/to/coverage_markers.yml
The above command run against the tests defined above would select ‘test_x’ and deselect ‘test_y’ for execution
Other examples of filters are:
You can also supply the path to a file containing your filter. Use argument –filter-location or key FilterLocation in the pytest.ini file.
Mandatory Coverage Markers
Coverage markers can be detailed as mandatory by including the mandatory attribute.
E.g.
markers:
- name: <marker_name>
mandatory: True
allowed:
- <marker_value_1>
- <marker_value_2>
Dependent Coverage Markers
Coverage markers can be detailed as a dependency on another marker. This ensures that if a marker is specified all dependencies of this marker in the chain must also be specified.
E.g.
markers:
- name: <marker_name>
dependents:
- <marker_name...>
- <marker_name...>
allowed:
- <marker_value_1>
- <marker_value_2>
Coverage Marker Argument Format
The arguments supplied to Coverage Markers can follow multiple formats which allows the user to define the format that best suites them.
E.g.
import pytest
@pytest.mark.marker_1('value1') # single string argument
@pytest.mark.marker_2('value1', 'value2') # multiple string arguments
@pytest.mark.marker_3(['value1', 'value2']) # list of arguments
@pytest.mark.marker_4(('value1', 'value2')) # tuple of arguments
def test_x():
...
Testing
Nox is used by this project to execute all tests. To run a specific set of tests execute the below line:
$ poetry run nox -s <session_name>
Where session_name can be one of the following
Session Name |
Session Details |
---|---|
unit_tests |
Execute all tests marked as unit |
functional_tests |
Execute all tests marked as functional |
Thought Process
The pytest_docs talks about using markers to set metadata on tests and use the markers to select required tests for execution.
For the markers I want to add, I also want to specify a list of values that go along with that marker. E.g. If the marker was ‘colour’ then supported values may be ‘Red’, ‘Green’, ‘Gold’.
I also want the list of values validated against supported values so no unsupported values can be added. E.g. If the marker was ‘colour’ then a value of ‘Panda’ would not be allowed.
Then all this meta data I want to come out in the junit json report.
Next I want to use these markers and their supported values to filter tests. For this I need a more powerful filter engine.
Documentation
To build the docs run:
poetry run mkdocs serve
License
Distributed under the terms of the MIT license, “pytest-coveragemarkers” is free and open source software
Issues
If you encounter any problems, please file an issue along with a detailed description.
Future Changes
Type-Hints
Full Test Coverage
Full Documentation
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pytest_coveragemarkers-3.2.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | b1472d6ab2a3c01bf08e8ecb63c7948564a4df715e65a5816ba62612227eba64 |
|
MD5 | d35720c952b4d36765a97b6c876b97f5 |
|
BLAKE2b-256 | ce6d0e0c137df140c4484bcab8321f6c0ba6cc9cf90cbe9bcf29f08bc4199e1b |
Hashes for pytest_coveragemarkers-3.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 03fe74980a855f66306ae3a1cabce4d4287a9838bcb7942e507ceb4f3c2c0840 |
|
MD5 | a97e76808c63acd63a76fd49c3a08adb |
|
BLAKE2b-256 | 763389a2839d97e8fc33b98ac9f7914979a3c6e18d27135a2df809b78b53e7ba |