uQCme - Microbial Quality Control Tool

A comprehensive quality control (QC) tool for microbial sequencing data that provides both command-line processing and interactive web-based visualization capabilities.

Overview

uQCme consists of two main components:

1. CLI Tool (uqcme): A command-line interface that processes microbial sequencing QC data against configurable quality control rules. It determines QC outcomes (PASS/FAIL/WARNING) based on species-specific criteria.
2. Web Dashboard (uqcme-dashboard): An interactive Streamlit application for visualizing and exploring the QC results generated by the CLI tool.

Features

Core Functionality

• Species-specific QC rules: Support for numerous microbial species with tailored quality control criteria defined in QC_rules.tsv.
• Configurable QC tests: Define custom QC outcomes with priority-based rule conditions.
• Flexible rule engine: Regex-based validation with threshold checks for various QC metrics.
• Robust Validation: Data validation using Pandera schemas and Pydantic configuration management.
• Interactive dashboard: Web-based visualization with filtering, sorting, and detailed sample exploration.
• Comprehensive logging: Detailed logging system with both file and console output.

Supported Species

The tool handles specific QC rules for the following species (as defined in the default QC_rules.tsv):

• Acinetobacter baumannii
• Campylobacter coli
• Campylobacter jejuni
• Enterobacter spp.
• Enterococcus faecium
• Escherichia coli
• Haemophilus influenzae
• Helicobacter pylori
• Klebsiella pneumoniae
• Mycoplasma genitalium
• Neisseria gonorrhoeae
• Pseudomonas aeruginosa
• Salmonella enterica
• Shigella flexneri
• Shigella sonnei
• Staphylococcus aureus
• Streptococcus pneumoniae

Note: "all" rules apply to any species not explicitly listed or as general baseline checks.

QC Metrics

• Assembly statistics (N50, contigs, genome size)
• CheckM completeness and contamination
• Species identification validation
• Coverage depth analysis
• Quality score assessments

Installation

From PyPI

Core only (shared logic):

pip install uqcme

CLI only:

pip install uqcme[cli]

Dashboard/App only:

pip install uqcme[app]

Full installation (CLI + Web Dashboard):

pip install uqcme[all]

From Source

git clone https://github.com/ssi-dk/uQCme.git
cd uQCme

# Core only
pip install .

# Full installation
pip install ".[all]"

Usage

1. CLI Tool (uqcme)

The CLI tool reads your sequencing run data and applies the QC rules defined in your configuration.

Basic Usage (using defaults):

uqcme

This will use the bundled default configuration and look for input files in the current directory as specified in the default config.

Override Data Source: You can process a specific data file or API endpoint without creating a full config file:

# Process a local file
uqcme --file path/to/my_run_data.tsv

# Process data from an API
uqcme --api-call "https://api.example.com/runs/123"

# Process data from an authenticated API (recommended: env var)
export UQCME_API_TOKEN="your-token-here"
uqcme --api-call "https://api.example.com/runs/123" \
  --api-bearer-token-env UQCME_API_TOKEN

Custom Configuration: For full control over rules, tests, and mappings, provide a custom configuration file:

uqcme --config my_config.yaml

What it does:

1. Loads run data (from file, API, or defaults).
2. Loads QC rules (QC_rules.tsv) and QC tests (QC_tests.tsv).
   • Note: If local rule files are missing, it falls back to bundled defaults.
3. Evaluates each sample against the rules for its species.
4. Determines the final QC outcome (e.g., PASS, FAIL).
5. Outputs a new TSV file containing the original data plus the QC results (e.g., qc_results.tsv).

2. Web Dashboard (uqcme-dashboard)

The dashboard visualizes the results generated by the CLI tool.

Command:

uqcme-dashboard --config config.yaml

Export dashboard report view as PDF (non-interactive):

uqcme-dashboard --config config.yaml --export-pdf output/report.pdf

This uses report-mode rendering from app.dashboard.report_mode in the config and exports a table-only report (filtered data dataframe). For PDF export support, install Playwright and Chromium once:

pip install playwright
playwright install chromium

What it does:

1. Launches a local web server (Streamlit).
2. Loads the processed data (from file or API) as specified in config.yaml.
3. Provides an interactive interface to:
   • View summary statistics.
   • Filter samples by QC outcome, species, or specific metrics.
   • Inspect individual sample details and failed rules.
   • Visualize metric distributions.

Configuration

The tool is driven by a config.yaml file. This file defines:

• Input paths: Locations of your data, mapping file, and QC rules/tests.
• Output paths: Where to save the results and logs.
• App settings: Dashboard configuration (server port, UI preferences).

Key Input Files:

• run_data.tsv: Your raw sequencing metrics.
• mapping.yaml: Maps your data columns to the tool's internal field names.
• QC_rules.tsv: Defines the specific thresholds and checks for each species. (default values originally from https://www.pathogensurveillance.net/resources/quality/ and https://happykhan.github.io/qualibact/)
• QC_tests.tsv: Defines how rule failures translate into overall QC outcomes.

Bearer token for api_call

The default config.yaml runs locally from synthetic files. Run the CLI first to generate its dashboard input. API input is opt-in; config.api.test.yaml provides a separate integration example.

When qc.input.data.api_call or app.input.data.api_call requires bearer auth, configure one of:

• api_bearer_token_env (recommended): environment variable name containing the token
• api_bearer_token: raw token in config file
• api_query_params: URL query params to forward (for example project_id)

app:
  input:
    data:
      api_call: "https://api.example.com/runs/123"
      api_bearer_token_env: "UQCME_API_TOKEN"
      api_query_params:
        - project_id
        - seqSamples
      # api_bearer_token: "plain-text-token"  # avoid in shared files

Config-driven sample API action buttons

You can configure one or more action buttons in the dashboard that send values from selected samples to an API endpoint:

app:
  dashboard:
    sample_api_actions:
      - label: "Notify LIMS"
        api_call: "https://example.org/api/notify"
        value_field: "sample_name"
        method: "POST"
        api_bearer_token_env: "UQCME_ACTION_TOKEN"
        payload_field: "sample_ids"
        send_as_list: true
        include_sample_ids: false

When users select samples in the Data Preview table and click the configured button, uQCme sends the selected value_field values to api_call. For authenticated button actions, configure one of:

• api_bearer_token_env (recommended): environment variable name containing the token
• api_bearer_token: raw token in config file
• headers: optional additional request headers

Dashboard table height

The main sample table uses table_height as a maximum height. For smaller datasets, the table shrinks to the rendered row count instead of leaving empty space. Increase or reduce the maximum visible row count with:

app:
  dashboard:
    table_height: 3600

Config-driven report mode defaults

Use the same dashboard config to control deterministic report rendering:

app:
  dashboard:
    report_mode:
      enabled: false
      default_visible_sections:
        Basic: true
        QC_metrics: true
        Experimental: false
      default_filters:
        species: "Escherichia coli"
        completeness:
          min: 90

See the input/example/ directory for template files.

Canonical species names

Species aliases belong in the mapping file, keyed by the name used in QC rules:

SpeciesAliases:
  Klebsiella pneumoniae:
    - K. pneumoniae

The QC engine resolves aliases for species-rule selection and species-value checks; the dashboard uses the same resolver. Original input values are retained. See species mapping and QC caveats for migration from the earlier alias-key format, alternate source columns and missing-metric flags.

Rule-aware species styling

The dashboard derives species support from explicit, species-specific entries in the configured QC rules. Rules targeting all do not mark a species as supported. Supported species use normal table styling, unsupported species use subtle blue text, and missing species use a translucent gray cell:

app:
  ui_styling:
    unsupported_species_color_light: "#1D4ED8"
    unsupported_species_color_dark: "#60A5FA"
    missing_species_color: "#808080"
    missing_species_opacity: 0.15

Colors must use six-digit hexadecimal notation (#RRGGBB), and opacity must be between 0 and 1. Species names are matched after trimming whitespace and without regard to case. Null, empty, or whitespace-only table values display as an em dash and appear as Missing species in the distribution chart.

Output Files

1. QC Results (qc_results.tsv)

Enhanced run data with QC outcomes:

• Original sample data
• Failed rules per sample
• Assigned QC outcome with priority
• Color coding for visualization

2. Rule Warnings (qc_warnings.tsv)

Detailed log of rule evaluation issues:

• Skipped rules and reasons
• Data validation warnings
• Processing statistics

Dashboard Features

Data Overview

• Interactive data table with filtering and sorting
• Priority-based color coding of QC outcomes
• Summary statistics and sample counts

Sample Details

• Config-driven navigation index of filtered samples
• With no extra settings, the index uses the mapped sample ID, species, QC outcome and QC action. app.dashboard.sample_details.index_columns optionally selects and orders source columns; report.label overrides their headings
• The field marked with report.id: true links to each sample's detail section
• Column presets are optional; omitting them leaves the basic table available
• Missing QC fields appear as per-sample caveats alongside the existing result
• Detailed view of individual sample QC results
• Failed rules and thresholds
• Interactive metric exploration

Visualization

• Plotly-based interactive charts
• Customizable metric comparisons
• Species-specific analysis views

Filtering and Search

• Dynamic filtering by QC outcome, species, and metrics
• Search functionality across all data columns
• Export capabilities for filtered datasets

Advanced Usage

Standalone Bundles

uQCme can build relocatable python-build-standalone bundles for deployment targets where installing Python packages directly is undesirable. Build Linux release artifacts on Linux so the embedded Python runtime matches the target host.

pixi run build-standalone

The build writes both profiles:

packaging/dist/uqcme-cli-standalone.tar.gz
packaging/dist/uqcme-dashboard-standalone.tar.gz

The CLI bundle exposes uqcme. The dashboard bundle exposes uqcme, uqcme-dashboard, and uqcme-dashboard-smoke. Release builds publish:

uqcme-cli-standalone-linux-x86_64.tar.gz
uqcme-dashboard-standalone-linux-x86_64.tar.gz

Use the smoke command to validate a dashboard runtime:

uqcme-dashboard-smoke --config config/config.yaml --port 49105

Custom QC Rules

Create custom QC rules by editing QC_rules.tsv:

rule_id	species	qc_tool	qc_metric	validation_type	threshold	column_name
CUSTOM1	Escherichia coli	Assembly	N50	threshold	>=50000	n50
CUSTOM2	all	CheckM	Completeness	threshold	>=90	completeness

Species-Specific Tests

Define new QC tests in QC_tests.tsv:

outcome_id	outcome_name	description	priority	passed_rule_conditions	failed_rule_conditions	action_required
FAIL_CUSTOM	Fail - Custom QC	Custom quality control failed	3		CUSTOM1,CUSTOM2	reject

Development

This project uses pixi for dependency management and development workflow.

To set up the development environment:

pixi install

To run tests:

pixi run pytest

Project Structure

uQCme/
├── src/
│   └── uQCme/
│       ├── __init__.py
│       ├── app/
│       │   ├── main.py     # Streamlit web dashboard entry
│       │   ├── plot.py     # Dashboard plotting utilities
│       │   └── smoke.py    # Dashboard startup smoke CLI
│       ├── cli/
│       │   └── main.py     # CLI entry
│       └── core/
│           ├── engine.py   # QC processing engine
│           ├── loader.py   # Config and data loading
│           └── config.py   # Pydantic config models
├── config.yaml             # Configuration file
├── input/
│   └── example/            # Example input files
├── output/                 # Generated results
├── log/                    # Application logs
└── tests/                  # Unit tests

Running Tests

python -m pytest tests/

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Submit a pull request

Troubleshooting

Common Issues

1. Missing input files: Ensure all required input files exist and paths in config.yaml are correct
2. Rule validation errors: Check that QC rules reference valid column names in your data
3. Dashboard not loading: Verify Streamlit installation and port availability

Logging

Check the log file (./log/log.tsv) for detailed processing information:

tail -f ./log/log.tsv

Citation

If you use uQCme in your research, please cite:

uQCme: A Comprehensive Quality Control Tool for Microbial Sequencing Data
SSI-DK, 2025

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

For questions, issues, or feature requests:

• Create an issue on GitHub
• Contact: Kim Ng (kimn@ssi.dk)

Changelog

See CHANGELOG.md for version history and updates.

Build and configuration identity

The dashboard displays the installed package version, a short source revision, and a presentation/QC configuration fingerprint. Payload version labels do not override the installed software version. Unknown provenance stays unknown; editable source checkouts report +dirty when modified. Standalone bundles embed the source revision, dirty state and UTC build datetime and retain them when moved.

uqcme --version and uqcme-dashboard --version exit without loading input data. --diagnostics prints safe identity JSON; add --config config/config.yaml from the normal working directory to include configuration identity. The dashboard's ?diagnostics=1 or app.dashboard.debug_diagnostics: true opens the same safe information in an expander; diagnostics=0 hides the details.

The configuration fingerprint covers presentation settings, source kind and mapping/QC reference bytes. It excludes input rows, paths, API endpoints, headers and credentials, and is not a complete installed-payload checksum or an API-authentication check. Compare complete configuration privately through the owning deployment's verification. Missing reference files set references_complete: false. Safe diagnostics never make an API request. The older ?debug=1 / debug_api feature is separate: it shows raw API diagnostic information and should only be used within the data access policy.

Deployments may supply an optional deployment.json beside the config/ directory containing schema_version: 1, flow_revision, installer_revision, flow_dirty, installer_dirty, runtime_id, environment, data_mode (synthetic, live, or unspecified) and artifact_sha256. Revisions use full 40-character Git IDs; archive digests use SHA-256. Unknown or invalid metadata is ignored rather than printed. This sidecar is deployment provenance, not proof of live health, completed deployment or successful external authentication.