Coverage-based regression test selection (RTS) plugin for pytest

• Usage
• Troubleshooting
• Development
• Contributing

Usage

Plugin is supposed to be used to execute tests related to changes done locally on developer's machine and in CI environment to test pull requests.

Initialization

To start using pytest-rts build of coverage DB is needed. For Trunk Based Development mapping database from master branch should be used, for A successful Git branching model - develop

1. Install pytest-cov with pip install pytest-cov
2. Create a .coveragerc file with the following contents inside to configure pytest-cov:

[run] 
relative_files = True

3. Execute pytest --cov=[path to your package] --cov-context=test --cov-config=.coveragerc which will run the entire test suite and build a mapping database in .coverage file
4. Rename the coverage file .coverage produced by pytest-cov to your liking. Example: mv .coverage pytest-rts-coverage

Note that --cov-config=.coveragerc is optional parameter here. For more info see official docs.

Local usage

Tests from changes in Git working directory

1. Install pytest-rts with pip install pytest-rts
2. Create a branch git checkout -b feat/new-feature
3. Make changes in your code
4. Run the tool with pytest --rts --rts-coverage-db=[path to database]

As a result only tests related to changes in working directory will be executed.

Tests from changes in Git working directory + committed changes

1. Install pytest-rts with pip install pytest-rts
2. Create a branch git checkout -b feat/new-feature
3. Make changes in your code and commit them
4. Run the tool with pytest --rts --rts-coverage-db=[path to database] --rts-from-commit=[database initialization commithash]

The current git working directory copy will be compared to the given commithash. Tests for changes in commits and in the working directory will be executed.

Usage in CI

• In the main branch (master or develop) make sure you run entire test suite and
  • commit back coverage database
  • or, if the database size is big, upload it to some storage
• In pull requests:
  • make sure you have coverage database from the main branch located next to the code
  • run pytest --rts --rts-coverage-db=[path to database] --rts-from-commit=[database initialization commithash]

One of the ways to organize it in Makefile would be:

BRANCH_NAME = $(shell git branch | sed -n -e 's/^\* \(.*\)/\1/p')
ifeq ($(BRANCH_NAME), master)
test: test-master
else
test: test-pr
endif

test-master: 
	pytest \
          --exitfirst \
          --cov \
          --cov-context=test \
          --cov-config=.coveragerc
	mv .coverage mapping.db
	git add mapping.db
	git commit -m "test: updated RTS mapping DB"
	git push

test-pr: MASTER_COMMIT = $(shell git merge-base remotes/origin/master HEAD)
test-pr:
	git checkout $(MASTER_COMMIT) mapping.db
	pytest \
          --exitfirst \
          --rts \
          --rts-coverage-db=mapping.db \
          --rts-from-commit=$(MASTER_COMMIT) && [ $$? -eq 0 ] || [ $$? -eq 5 ]

Exit code tests/overwrite && [ $$? -eq 0 ] || [ $$? -eq 5 ] is needed in cases when no tests are found for execution. See Troubleshooting section for more information.

Usage with coverage threshold

You might desire to use pytest-cov with the --cov-fail-under=MIN flag. When using pytest-rts this is somewhat possible but the reported coverage percentage will not represent actual coverage. If you wish to combine the usage of --rts and the coverage threshold, do the following:

1. Initialize the coverage database as usual
2. Leave the .coverage file untouched or rename it to start with .coverage.. For example .coverage.rts.db
3. Do your changes and add --cov=[path to code] and --cov-append to the flags. For example pytest --rts --rts-coverage-db=.coverage --cov=. --cov-append
4. pytest-cov will automatically combine the coverage file for the previous full run and the current RTS run. Bear in mind that the coverage file will now have added data. Create a backup of your full test run coverage file if you wish to keep it intact.

Troubleshooting

• pytest --rts returns non-zero code: command returns one of the pytest exit codes. For example if pytest-rts module found no tests to execute resulting code will be 5 "No tests were collected"

Development

See DEVELOPER.md for more info

Contributing

Contributing Guidelines

Read through our contributing guidelines to learn about our submission process, coding rules and more.

Code of Conduct

Help us keep the project open and inclusive. Please read and follow our Code of Conduct.

Acknowledgement

The package was developed by F-Secure Corporation and University of Helsinki in scope of IVVES project. This work was labelled by ITEA3 and funded by local authorities under grant agreement “ITEA-2019-18022-IVVES”