Github actions plugin to check flakiness of tests by calculating fliprates.
Project description
flaky_tests_detection
Github actions plugin to check flakiness of tests by calculating fliprates. Usage demonstrated here at the actions
page.
Features
- Prints out top test names and their latest calculation window scores (normal fliprate and exponentially weighted moving average fliprate that take previous calculation windows into account)
- Calculation grouping options:
n
daysn
runs
- Heatmap visualization of the scores and history
Parameters
Data options (choose one)
--test-history-csv
- Give a path to a test history csv file which includes three fields:
timestamp
,test_identifier
andtest_status
- Give a path to a test history csv file which includes three fields:
--junit-files
- Give a path to a folder with
JUnit
test results
- Give a path to a folder with
Calculation options
-
--grouping-option
days
to usen
days for fliprate calculation windowsruns
to usen
runs for fliprate calculation windows
-
--window-size
- Fliprate calculation window size
n
- Fliprate calculation window size
-
--window-count
- History size for exponentially weighted moving average calculations
-
--top-n
- How many top highest scoring tests to print out
Heatmap generation
--heatmap
- Turn heatmap generation on
- Two pictures generated: normal fliprate and exponentially weighted moving average fliprate score
- Same parameters used as with the printed statistics
Full examples
- Precomputed
test_history.csv
with daily calulations. 1 day windows, 7 day history and 5 tests printed out--test-history-csv=example_history/test_history.csv --grouping-option=days --window-size=1 --window-count=7 --top-n=5
JUnit
files with calculations per 5 runs. 15 runs history and 5 tests printed out--junit-files=example_history/junit_files --grouping-option=runs --window-size=5 --window-count=3 --top-n=5
- Precomputed
test_history.csv
with daily calculations and heatmap generation. 1 day windows, 7 day history and 50 tests printed and generated to heatmaps.--test-history-csv=example_history/test_history.csv --grouping-option=days --window-size=1 --window-count=7 --top-n=50 --heatmap
Install module
make install
Install module and development packages
make install_dev
Run pytest
make run_test
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for flaky-tests-detection-1.0.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 229a2e63dfb96ecf764f03fa2a2bdfa0bbbdb9ffb54ca0a755d84b45f4e85843 |
|
MD5 | 70a347ff26b72972bc4b8cde6b32d3d4 |
|
BLAKE2b-256 | 27aaa4be51385d0d3c9336d7473087e17ab823b9084ceb8182c9daac0d110d09 |
Close
Hashes for flaky_tests_detection-1.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bec41c849d558c39118ba649eb5223efa96e6d46c72ebf9278c7dd57480f942d |
|
MD5 | 4f015dbb370bb5d4111e01525757622e |
|
BLAKE2b-256 | e053caf0987773f385840d2131259dbc9a9defb186846933f4bef06a6aa72882 |