genomic-benchmarks-qc 1.0.0

Find the shortcut. Learn the biology.

Automated quality control for genomic machine learning datasets: scores the biases, duplicates and data leakage a classifier could exploit before you train on it.

Everything you need is below. The documentation site adds what a README has no room for: eight worked examples with live reports — a clean dataset, a badly biased one, one whose only flaw is six positions wide — a guide to what to do about each check and why the thresholds sit where they do, and the generated CLI and API reference.

What it catches

When the classes differ in something trivial, a high score no longer tells you what a model learned — the biology, or the shortcut. gb-qc looks for the differences a classifier could exploit without understanding anything:

• Your negatives are shorter than your positives. A length classifier now beats your model.
• Your classes differ in GC content, base composition or dinucleotide frequencies. In the enhancers example, GC content alone separates the two classes at AU-ROC 0.66.
• Position 1 is N in one class only, or every sequence in one class starts with the same adapter — a per-position give-away that no summary statistic would show.
• The same sequence appears in both classes, or your test set repeats your training set. evaluate-splits catches near-duplicates too, with an MMseqs2 similarity search.

Each check gets a Pass / Warning / Fail flag, an HTML report you can read, and a CSV you can put in CI.

The report for the composition-bias example, scrolled end to end — click it to open the real one. It is the worst case in the gallery, and the one report that carries all three flags at once: six checks Fail — the classes share sequences outright, and GC content, nucleotide and dinucleotide composition and both per-position checks all sit above the 0.7 line, the worst at AU-ROC 0.717 — one Warning, two Pass.

Installation

pip install genomic-benchmarks-qc

Requires Python 3.12 or newer.

Quick Start

Point the tool at your dataset and give it somewhere to write. To run the commands below exactly as they stand — and compare what you get against the reports they link to — fetch the two enhancers example files first:

curl -fLO https://raw.githubusercontent.com/genomic-benchmarks/genomic-benchmarks-qc/main/examples/enhancers/data/enhancers_train.csv
curl -fLO https://raw.githubusercontent.com/genomic-benchmarks/genomic-benchmarks-qc/main/examples/enhancers/data/enhancers_test.csv

gb-qc evaluate-classes \
  --input enhancers_train.csv \
  --input enhancers_test.csv \
  --out-folder qc-out

It leaves you a report to open — this one:

qc-out/class/sequence/0_vs_1/
├── gb-qc-report.html   ← open this
├── gb-qc-report.csv
└── plots/

To check whether your test set leaks into your training set:

gb-qc evaluate-splits \
  --train-input enhancers_train.csv \
  --test-input enhancers_test.csv \
  --sequence-column sequence \
  --out-folder qc-out

with a report of its own — this one:

qc-out/split/sequence/enhancers_train_vs_enhancers_test/
├── gb-qc-report.html   ← open this
├── gb-qc-report.csv
└── plots/

Both commands can share one --out-folder; they write into class/ and split/ respectively and never overwrite each other.

Reading the report

The HTML report is a single standalone file — no external assets, so you can mail it or archive it. It opens with a summary table of every check and its flag, followed by the basic descriptive statistics of each class, then one section per check with the numbers and the plot behind it.

What it checks:

• Nucleotide vocabulary
• Sequence lengths distribution
• GC content per sequence
• Nucleotide composition (per sequence & per position)
• Dinucleotide frequencies
• Sequence duplication levels
• Exact duplicate sequences between classes

Flags come from the AU-ROC of a classifier that sees only that one feature:

Unknown is not Pass: it says the comparison was not made, not that it came out clean. A check needs at least 250 sequences before it is scored at all. The plots, per-class statistics and descriptive tables are computed from all the data either way, so small datasets can still be compared by eye, and the terminal says which checks were skipped and why.

The per-position plot is interactive

Full walkthrough: the per-position plot.

For long sequences a static plot cannot show much: a single flagged position is lost in a figure spanning hundreds of them. The per-position panels are therefore interactive, so you can zoom in and explore the individual failing positions.

• Drag to zoom, shift-drag to pan, double-click to reset.
• Hover for the per-class base frequencies and flags at a position.
• Prev / Next flag jumps to the flagged positions.
• Save view writes the current window as a PNG.
• The table underneath lists every flagged position and jumps the plot to it.

With variable-length sequences, the tail positions that too few sequences reach are not scored and are left out of the figures.

The leakage report

Full treatment, including how similarity is computed: train/test leakage.

evaluate-splits searches every test sequence against the training set with MMseqs2 and flags one when it exceeds --similarity-threshold (90% by default). The report gives the percentage of leaked queries and targets, a histogram of the similarity distribution, and a panel listing the leaked pairs — up to the first 100, each expanding into its rendered alignment, so you can see what is actually shared. Every flagged pair, listed or not, is exported to mmseqs/mmseqs2_search_result.tsv beside the report. The enhancers example carries a little real leakage: 0.67% of queries and 0.33% of targets — see the report.

Input Formats

All of the input formats are supported in .gz version as well.

The commands in this section name files from the repository's examples/ directory to show the option shapes; substitute your own paths.

CSV/TSV needs a sequence column and a label column. By default they are called sequence and label:

sequence,label
GAGTGTATGTGTCGAGGAATGTATCCATTTCTTCTAGATTTTCTAGTTT,1
GGTCACCACCACCAAGTTCATGCCTGAACCCTTCAGTGGTCCTTTGCCC,0

Override the names with --sequence-column and --label-column. Any other columns are ignored.

FASTA files carry no labels, so each file is one class and its filename stem (the name without extensions) becomes the label — coding_seqs.fasta becomes the label coding_seqs:

gb-qc evaluate-classes \
  --input examples/fasta-classes/data/coding_seqs.fasta \
  --input examples/fasta-classes/data/intergenomic_seqs.fasta \
  --out-folder qc-out

Multiple sequence columns, for datasets where one row pairs two sequences: evaluate-classes analyzes each column separately, then concatenates all sequences for a combined merged analysis. evaluate-splits only analyzes concatenated sequences.

gb-qc evaluate-classes \
  --input examples/paired-sequences/data/miRNA_mRNA_pairs_dataset.tsv \
  --sequence-column gene \
  --sequence-column noncodingRNA \
  --label-column label \
  --out-folder qc-out

Continuous labels: --regression splits the label column at its median into high and low classes. Rows whose value is not numeric are dropped with a warning, and if the split leaves only one class — a constant column, or one that is mostly zeros — the run exits with an error.

Repeat the option name for each value. --input a.csv --input b.csv, not --input a.csv b.csv. The same goes for --sequence-column and --label-list.

Output Files

Each command writes into its own sub-directory of --out-folder — class/ for evaluate-classes and split/ for evaluate-splits — then into one directory per sequence column, then into one directory per comparison:

<out-folder>/
├── class/
│   └── sequence/
│       └── negatives_vs_positives/
│           ├── gb-qc-report.csv
│           └── gb-qc-report.html
└── split/
    └── sequence/
        └── train_vs_test/
            ├── gb-qc-report.csv
            └── gb-qc-report.html

If you are checking several datasets, give each one its own --out-folder to keep the results side by side.

<out-folder>/class/
└── <column>/                   # one directory per sequence column, named after
    │                           # it. FASTA inputs have no columns and use
    │                           # `sequence`; if you give several columns, an
    │                           # extra `merged` report joins them all together
    ├── <classA>_vs_<classB>/
    │   ├── gb-qc-report.csv
    │   ├── gb-qc-report.html
    │   ├── gb-qc-duplicates.txt
    │   └── plots/
    └── per-class/
        └── <class>.json

<out-folder>/split/
└── <column>/                   # the sequence column that was searched. FASTA
    └── <train>_vs_<test>/      # inputs use `sequence`, and several columns
        ├── gb-qc-report.csv    # searched together use `merged`
        ├── gb-qc-report.html
        ├── plots/
        └── mmseqs/

CLI Reference

Run gb-qc evaluate-classes --help or gb-qc evaluate-splits --help for the same thing in your terminal.

Contributions & Support

Contributions and suggestions for new features are welcome, as are bug reports! Please create a new issue for any of these, including example reports where possible. Pull-requests for fixes and additions are very welcome. See the contributing notes for more information about how the process works.

Citation

If you use genomic-benchmarks-qc in your research, please cite this repository: https://github.com/genomic-benchmarks/genomic-benchmarks-qc

License

MIT-style. See LICENSE.