fastq-sheet-audit

A local, offline FASTQ ↔ sample-sheet preflight tool with a command-line audit, a native desktop GUI, and explicit, validated sample-sheet export.

This README describes fastq-sheet-audit v0.2.0. The historical v0.1 prototype is preserved in Git under the v0.1.0 tag.

Why this exists

A sequencing workflow can fail before analysis begins because a sheet points to missing files, mixes lanes or chunks, assigns the same FASTQ twice, or leaves files unaccounted for. fastq-sheet-audit makes those discrepancies visible before a downstream pipeline runs. It separates discovered evidence, automatic interpretation, and explicit human decisions.

Capabilities and design

• Lossless CSV/TSV import, deterministic column mapping, and recursive FASTQ inventory.
• Structural filename parsing, exact mate identities, and explicit read-layout checks.
• Findings for missing/reused files, role/sample/pair disagreements, and unlisted files.
• Case-portability diagnostics and GUI pairing adjudication without destroying evidence.
• Declarative export profiles, manual metadata editing, path previews, and CSV/TSV exports.
• Structured JSON audit reports for machine consumers.

The application is fully local/offline: no telemetry, cloud/API dependency, or AI/LLM dependency. It uses filename structure and filesystem metadata; FASTQ sequence contents are never opened or read. Interpretation is deterministic, with no fuzzy matching or biological inference from filenames. Ambiguity stays visible until an explicit decision is made.

FASTQs and input sample sheets are never renamed, moved, deleted, repaired, or modified. Auditing leaves inputs untouched; explicit report/export actions can create output files. Publication protects the input sheet, raw discovered FASTQs, and referenced FASTQ paths, including references outside the scan root and missing referenced destinations. Resolved aliases and existing symlink/hardlink aliases are checked too.

Outputs use a temporary file in the destination directory, UTF-8 validation, flush/fsync, and atomic replacement. Overwrite is conservative: CLI reports require --overwrite-report to replace an ordinary file; GUI exports currently refuse existing destinations. Protection remains active even with report overwrite. No-overwrite publication reserves the destination exclusively before replacement. An empty reservation can briefly be visible. Python's portable APIs do not offer an atomic conditional replace, so hostile concurrent directory/path changes or replacement of that reservation cannot be fully guarded against. This is best-effort race safety, not a guarantee against concurrent filesystem mutation.

Installation

Python 3.10 or newer is required. From a checkout of the v0.2 development branch:

python -m pip install .
fastq-sheet-audit --version

The GUI uses standard-library tkinter/ttk and needs an available Tk installation and desktop display. Some Python distributions provide Tk separately. CLI use does not require opening the GUI. Dependency installation may use the network; audits and exports do not.

CLI quick start

Input sheets may be UTF-8 CSV or TSV, including a UTF-8 BOM, quoted fields, LF, and CRLF. Headers, column order, cell text, and physical row numbers are preserved. Duplicate headers after surrounding-whitespace trimming, surplus fields, malformed quoting, and missing headers are rejected. Short rows receive empty trailing cells; only entirely empty rows are skipped.

Example samples.csv:

sample,r1,r2
A,A_S1_L001_R1_001.fastq.gz,A_S1_L001_R2_001.fastq.gz
A,A_S1_L002_R1_001.fastq.gz,A_S1_L002_R2_001.fastq.gz

fastq-sheet-audit check samples.csv --fastq-dir ./fastq
fastq-sheet-audit check samples.csv --fastq-dir ./fastq --read-mode paired --json audit.json

Relative FASTQ references resolve against --fastq-dir, including subdirectories, not against the sheet's directory. Absolute references remain absolute. Path cell text is used without trimming, variable expansion, or rewriting. Discovery recognizes .fastq.gz, .fq.gz, .fastq, and .fq, case-insensitively. Directory symlinks are not traversed recursively; eligible file symlinks may be inventoried.

The CLI prints a deterministic summary and ordered findings. Available options:

Column mapping

Automatic mapping compares headers using surrounding-whitespace trimming and Unicode casefold only. The deterministic aliases are:

SAMPLE and R1 are required; R2 is optional. Multiple candidates for any role are refused unless explicitly resolved, including an ambiguous optional R2. Errors list physical column numbers and exact headers. Overrides can select non-alias headers; omitted overrides retain automatic mapping. One physical column cannot fill multiple roles. R2 selection and unmapping are mutually exclusive.

For a sheet with specimen,forward,reverse:

fastq-sheet-audit check samples.csv --fastq-dir ./fastq --sample-column 1 --r1-column 2 --r2-column 3

The GUI's Load columns action offers indexed choices, Automatic, and Unassigned. Leaving SAMPLE or R1 unassigned prevents auditing. Unknown metadata columns survive import; profile exports contain the selected profile's columns rather than automatically copying arbitrary source metadata.

Read modes, pairing, and findings

Index reads, Undetermined files, and unparsed names remain visible and are excluded from biological layout inference. The parser recognizes common forms such as A_R1.fastq.gz, A_1.fastq, A_S1_L001_R1_001.fastq.gz, and A_I1_001.fastq.gz; it does not claim exhaustive naming support.

Mate identity includes sample, sample number, lane, chunk, read style, suffix, and relative parent directory. Only sample and suffix use Unicode casefold. Lane/sample-number/chunk differences and R-style versus bare 1/2 remain significant. Directory identity preserves exact spelling through a platform-independent representation: Run/A_R1.fastq and run/A_R2.fastq are separate identities. Files are never paired by list position or proximity.

Repeated sample IDs are allowed when rows use distinct FASTQ evidence, as in the two-lane example. Actual file reuse, including filesystem aliases, is an error (FASTQ_REUSED). Sample-versus-filename comparison trims surrounding sheet sample whitespace and uses Unicode casefold only: A-B and AB remain distinct. Original text is retained.

An omitted optional R2 is not automatically assigned. If an exact R2 exists on disk but is absent from the sheet, it remains UNLISTED_FASTQ evidence. Unknown/unparsed filenames are retained, not guessed. Filename role disagreements, structural pair mismatches, and missing references are reported. Case-only relative-path collisions are warnings; this is a case-portability check, not a complete set of Windows filename rules.

GUI quick start and adjudication

fastq-sheet-audit-gui

Choose a FASTQ directory and sheet, load/resolve columns if needed, select a read mode, and press Audit. Opening the GUI does not scan inputs automatically. Summary, Findings, FASTQ Inventory, and Pairing / Adjudication tabs expose the current evidence.

Select a pair row to choose R1/R2 candidates explicitly, leave a role Unassigned, or Reset automatic. Confirmed records human confirmation; it does not resolve ambiguity or override errors. Applying a decision revalidates the workflow. All original candidates remain in the evidence even after a selection or unassignment. Unresolved automatic choices remain unresolved.

Read-layout checks use effective adjudicated reads, while reconciliation and case-collision checks retain raw inventory. Decisions cannot suppress unrelated findings. The tool does not edit the source sheet to repair discrepancies; correct it externally and rerun Audit when necessary. Changed inputs require a new audit, and changed export options require a fresh preview.

Export profiles and manual metadata

GUI export requires a clean workflow, followed by a valid profile preview. Choose a profile and path mode, set any needed manual fields per source row, then press Preview export. Choose CSV or TSV and a destination explicitly before pressing Export. Format is independent of filename extension. The CLI audits and publishes JSON reports; it does not export sample sheets.

Export uses exact source sample text and effective adjudicated R1/R2 records, not the original sheet's FASTQ path cells after adjudication. Profile columns declare their source roles. Columns with no source role are manual: no metadata is inferred and no descriptive profile defaults are automatically inserted. Set supplies exact text, including an explicit empty string; Clear makes the value absent. Unapplied editor text must be applied or cleared before preview/export.

Validation distinguishes a required column from a required cell. String allowed values are matched exactly without trimming or casefolding. Integer values use an optional sign and ASCII decimal digits; text such as 01 remains unchanged in output. Whitespace, Unicode, punctuation, and formula-like text are preserved; CSV/TSV export is not a spreadsheet-sanitization step.

Bundled profiles are local declarative contracts, not a guarantee that every pipeline option or future release is supported:

All five FASTQ-samplesheet profiles support single-end input. For rnaseq, methylseq, and viralrecon Illumina, fastq_2 is a required column but may contain empty cells. Generic and smrnaseq omit their optional R2 column when no row supplies an effective R2. Methylseq's genome column is required but its value is optional. Viralrecon's downstream sample-name rewriting is not reproduced here: sample IDs are not silently renamed or normalized.

Export path modes

Rebasing requires an explicit POSIX or Windows style and an absolute root in that style. It uses pure path transformations without accessing the target filesystem. Spaces and Unicode remain intact; unsafe relative components are rejected rather than resolved away.

JSON reports and exit codes

fastq-sheet-audit check samples.csv --fastq-dir ./fastq --json audit.json
fastq-sheet-audit check samples.csv --fastq-dir ./fastq --json audit.json --overwrite-report

JSON reports use schema_version: 1, independently of the package version. They preserve structured inventory, reconciliation findings and assignments, read-mode diagnostics and evidence, case collisions, and pair candidates, decisions, effective/unresolved records, and confirmation. Pair keys remain structured, absent values stay null, and Unicode remains literal. Ordering is deterministic. A report can be written even when the audit has findings; a publication failure instead returns exit 2.

Unexpected programmer errors are not converted into ordinary audit failures.

Scope and development

This is a filename/path/sample-sheet preflight tool. It does not inspect read contents, compare read counts, validate checksums, demultiplex, repair FASTQs, run pipelines, infer biological metadata, or provide clinical validation. Profile exports cover only the bundled declarative fields. CLI human pairing adjudication and barcode-workflow export are outside the current scope.

python -m pip install -e ".[dev]"
python -m pytest -q

Tests are headless; they do not launch Tk. CI covers Ubuntu Python 3.10–3.14, representative Python 3.12 jobs on Windows/macOS, and isolated sdist/wheel, entry-point, and profile-resource validation. See CHANGELOG.md for the v0.2.0 changes and historical 0.1.0 entry, and SECURITY.md for security reporting.