regression-testing-utils 0.1.2

Collection of Python tools to support execution of regression testing.

Regression Testing Utils

Collection of Python tools to support execution of regression testing.

Exported Console Scripts

The following exported console scripts are available:

• regression-testing-runner

Usage

Install the package:

pip install regression-testing-utils

Prepare your regression configuration file.

Contents of example.yaml:

---
# Acceptable values for file_format are csv or tsv.
# Meaning, this software can only compare files
# where both files are tab-delimited (tsv) or both
# files are comma-separated (csv).
file_format: tsv

# The line number that the column headers occur on.
# Column headers are necessary for reporting which
# which columns the differences were detected in.
header_line_number: 1

# The line number that the records start on.
# Beginning on this line, records will be compared
# between the two files.
records_start_line_number: 2

# The maximum number of columns.  If specified, this
# will inform the software up to which line should
# be consider. Otherwise, all lines beginning at-
# and including records_start_line_number (above).
max_column_count:

# If the following is True, then the evaluator software will ignore
# the columns listed in the "ignore_columns_str".
ignore_columns: false

# The following will list the columns to be ignored by the evaluator
# software.  Note: if the evaluator software should ignore columns
# specified in this list, then the "ignore_columns" boolean above
# should be set to true.
ignore_columns_str: null

workflow:
  # If running in test mode, the actual commands will not be executed.
  test_mode: false
  stages:
    # Each workflow stage is optional.
    # The stage_name and commands are all required.
    # The desc (description) is optional.
    - stage_name: evaluation
      desc: "Evaluate the results of the software being tested/evaluated"
      commands:
        - evaluator:
          # Set-up for testing:
          # cp regression-testing-utils/examples/example_actual.tsv /tmp/demo-regression-testing-utils/actual.tsv
          # cp regression-testing-utils/examples/example_expected.tsv /tmp/demo-regression-testing-utils/expected.tsv
          args:
            # file1 is required.  It must indicate the first file that will be compared against file2.
            file1: /tmp/demo-regression-testing-utils/actual.tsv
            # file2 is required.  It must indicate the first file that will be compared against file1.
            file2: /tmp/demo-regression-testing-utils/expected.tsv
            # file_format is optional.  It indicates what the file formats are.
            file_format: tsv
            # outfile is required.  It indicates where the comparison report will be written to.
            outfile: /tmp/demo-regression-testing-utils/comparison_report.txt
# If the following is set to true,
# the software will use the Rich Table
# API for displaying which stage is
# being executed.
# If set to false, will use a simple
# banner to display the same information.
use_rich_table: true

Contents of actual.tsv

cat -n /tmp/demo-regression-testing-utils/actual.tsv
1  #CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO
2  1       12345   rs567   A       G       50      PASS    DP=30;AF=0.2;AN=1000;CSQ=missense_variant|HIGH|GeneA|ENSG00000112345|transcriptA|ENST00000234567|protein_coding|1/10|c.123C>T|p.Arg41Trp|123/1000|ensembl
3  2       56789   rs555   T       C       44      PASS    DP=25;AF=0.1;AN=1200;CSQ=synonymous_variant|MEDIUM|GeneB|ENSG00000123456|transcriptB|ENST00000345678|protein_coding|5/20|c.567A>G|p.Ala189Ala|567/1200|ensembl
4  3       98765   rs123   G       T       61      PASS    DP=40;AF=0.3;AN=800;CSQ=splice_acceptor_variant|HIGH|GeneC|ENSG00000134567|transcriptC|ENST00000456789|protein_coding|2/15|c.987+1G>T|p.?|987/800|ensembl
5  1       34567   rs456   C       A       55      FAIL    DP=35;AF=0.15;AN=900;CSQ=frameshift_variant|HIGH|GeneX|ENSG00000145678|transcriptX|ENST00000567890|protein_coding|8/25|c.345_346insT|p.Leu116Phefs*12|345/900|ensembl

Contents of expected.tsv

cat -n /tmp/demo-regression-testing-utils/expected.tsv
1  #CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO
2  1       12345   rs567   A       G       50      PASS    DP=30;AF=0.2;AN=1000;CSQ=missense_variant|HIGH|GeneA|ENSG00000112345|transcriptA|ENST00000234567|protein_coding|1/10|c.123C>T|p.Arg41Trp|123/1000|ensembl
3  2       56789   rs890   T       C       44      PASS    DP=25;AF=0.1;AN=1200;CSQ=synonymous_variant|MEDIUM|GeneB|ENSG00000123456|transcriptB|ENST00000345678|protein_coding|5/20|c.567A>G|p.Ala189Ala|567/1200|ensembl
4  3       98765   rs123   G       T       60      PASS    DP=40;AF=0.3;AN=800;CSQ=splice_acceptor_variant|HIGH|GeneC|ENSG00000134567|transcriptC|ENST00000456789|protein_coding|2/15|c.987+1G>T|p.?|987/800|ensembl
5  1       34567   rs456   C       A       55      PASS    DP=35;AF=0.15;AN=900;CSQ=frameshift_variant|HIGH|GeneX|ENSG00000145678|transcriptX|ENST00000567890|protein_coding|8/25|c.345_346insT|p.Leu116Phefs*12|345/900|ensembl

Execution of exported console script:

regression-testing-runner --config_file /tmp/demo-regression-testing-utils/simple.yaml --outdir /tmp/demo-regression-testing-utils
--logfile was not specified and therefore was set to '/tmp/demo-regression-testing-utils/runner.log'
Setting file format to 'tsv'
3 differences found

Contents of comparison report:

cat -n /tmp/demo-regression-testing-utils/comparison_report.txt
 1  ## method-created: /tmp/regression-testing-utils/venv/lib/python3.10/site-packages/regression_testing_utils/evaluator.py
 2  ## date-created: 2025-02-15-121022
 3  ## created-by: sundaram
 4  ## file 1: /tmp/demo-regression-testing-utils/actual.tsv
 5  ## file 2: /tmp/demo-regression-testing-utils/expected.tsv
 6  ## file-format: tsv
 7  ## logfile: /tmp/demo-regression-testing-utils/runner.log
 8  ## Number of differences: 3
 9  Line #  Column Name     Column #        Column Letter   Value in File 1 Value in File 2
10  1       rs567   3       C       rs555   rs890
11  2       50      6       F       61      60
12  3       PASS    7       G       FAIL    PASS

History

0.1.0 (2024-02-24)

• First release on PyPI.