Github actions plugin to check flakiness of tests by calculating fliprates.
Project description
flaky_tests_detection
Github actions plugin to check flakiness of tests by calculating fliprates. Usage demonstrated here at the actions
page.
Features
- Prints out top test names and their latest calculation window scores (normal fliprate and exponentially weighted moving average fliprate that take previous calculation windows into account)
- Calculation grouping options:
n
daysn
runs
- Heatmap visualization of the scores and history
Parameters
Data options (choose one)
--test-history-csv
- Give a path to a test history csv file which includes three fields:
timestamp
,test_identifier
andtest_status
- Give a path to a test history csv file which includes three fields:
--junit-files
- Give a path to a folder with
JUnit
test results
- Give a path to a folder with
Calculation options
-
--grouping-option
days
to usen
days for fliprate calculation windowsruns
to usen
runs for fliprate calculation windows
-
--window-size
- Fliprate calculation window size
n
- Fliprate calculation window size
-
--window-count
- History size for exponentially weighted moving average calculations
-
--top-n
- How many top highest scoring tests to print out
Heatmap generation
--heatmap
- Turn heatmap generation on
- Two pictures generated: normal fliprate and exponentially weighted moving average fliprate score
- Same parameters used as with the printed statistics
Full examples
- Precomputed
test_history.csv
with daily calulations. 1 day windows, 7 day history and 5 tests printed out--test-history-csv=example_history/test_history.csv --grouping-option=days --window-size=1 --window-count=7 --top-n=5
JUnit
files with calculations per 5 runs. 15 runs history and 5 tests printed out--junit-files=example_history/junit_files --grouping-option=runs --window-size=5 --window-count=3 --top-n=5
- Precomputed
test_history.csv
with daily calculations and heatmap generation. 1 day windows, 7 day history and 50 tests printed and generated to heatmaps.--test-history-csv=example_history/test_history.csv --grouping-option=days --window-size=1 --window-count=7 --top-n=50 --heatmap
Install module
make install
Install module and development packages
make install_dev
Run pytest
make run_test
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for flaky-tests-detection-1.0.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 689f316ffe9d6288066d4c18cb7781e8473c1e5e99572837adbce044ef487738 |
|
MD5 | 0137b56b7e89352742de6b027a57caf4 |
|
BLAKE2b-256 | 07114ea6309e898adfb3c12f97d30c6d2115182df7ef1e69cfc589efabdd3781 |
Close
Hashes for flaky_tests_detection-1.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec3daa4300d584141fa95f10f01702755ace2c9c7903b0baaf303e2d1f66b0cc |
|
MD5 | 1cae22a7d6787dccecd5ba2a3027f7ff |
|
BLAKE2b-256 | 8eb52dce6cee111ab444721b747bdd2974d9b032b2f653b9fb08733060678d38 |