Github actions plugin to check flakiness of tests by calculating fliprates.
Project description
flaky_tests_detection
Github actions plugin to check flakiness of tests by calculating fliprates. Usage demonstrated here at the actions
page.
Features
- Prints out top test names and their latest calculation window scores (normal fliprate and exponentially weighted moving average fliprate that take previous calculation windows into account)
- Calculation grouping options:
n
daysn
runs
- Heatmap visualization of the scores and history
Parameters
Data options (choose one)
--test-history-csv
- Give a path to a test history csv file which includes three fields:
timestamp
,test_identifier
andtest_status
- Give a path to a test history csv file which includes three fields:
--junit-files
- Give a path to a folder with
JUnit
test results
- Give a path to a folder with
Calculation options
-
--grouping-option
days
to usen
days for fliprate calculation windowsruns
to usen
runs for fliprate calculation windows
-
--window-size
- Fliprate calculation window size
n
- Fliprate calculation window size
-
--window-count
- History size for exponentially weighted moving average calculations
-
--top-n
- How many top highest scoring tests to print out
Heatmap generation
--heatmap
- Turn heatmap generation on
- Two pictures generated: normal fliprate and exponentially weighted moving average fliprate score
- Same parameters used as with the printed statistics
Full examples
- Precomputed
test_history.csv
with daily calulations. 1 day windows, 7 day history and 5 tests printed out--test-history-csv=example_history/test_history.csv --grouping-option=days --window-size=1 --window-count=7 --top-n=5
JUnit
files with calculations per 5 runs. 15 runs history and 5 tests printed out--junit-files=example_history/junit_files --grouping-option=runs --window-size=5 --window-count=3 --top-n=5
- Precomputed
test_history.csv
with daily calculations and heatmap generation. 1 day windows, 7 day history and 50 tests printed and generated to heatmaps.--test-history-csv=example_history/test_history.csv --grouping-option=days --window-size=1 --window-count=7 --top-n=50 --heatmap
Install module
make install
Install module and development packages
make install_dev
Run pytest
make run_test
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for flaky-tests-detection-1.0.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 34cd6deabb8f0673b62770fa1e4fcc8e6650c481b88c53f8f8c2582081bb2918 |
|
MD5 | 61f6dce3dd3d53f9b4eb64f4fa5a4130 |
|
BLAKE2b-256 | ea7d2ab12ca78d7c9e67376fa76a93be0004c6c15eaf2adcb335c137a65da790 |
Close
Hashes for flaky_tests_detection-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2be0af922388ec2361510182a2c9f3992ad038d2993573e6762d629bf4bb4680 |
|
MD5 | 8fbb302d5bc3651b167b195444d915db |
|
BLAKE2b-256 | bb8f7fb143b625e9ed79ccf23cacf6124f7d80ec6dfbc3c929964d516c4d00b3 |