iobrx

Fast, faithful tumor microenvironment analysis with a pandas API.

English · 中文说明 · Tutorial gallery · CPU compatibility · Benchmarks

iobrx is an independently maintained acceleration and integration layer built on the original IOBRpy toolkit. It depends on IOBRpy and reuses its reference resources, gene signatures and analysis semantics. iobrx adds Rust kernels, parallel/vectorized execution, a pandas API, tested tutorials and Omicos agent interfaces. Inputs and outputs are ordinary DataFrames and files. Numerical parity is tested per method and fixture; it is not a universal bit-exact guarantee for every new workflow, parameter or input. See the explicit exceptions and validation scope.

Original IOBRpy repository · Official IOBRpy documentation

29 public APIs — 27 workflow functions (plus the deconvolute_quantiseq alias and the load_official data helper) covering the following IOBRpy workflow families: immune deconvolution (CIBERSORT, BayesPrism, EPIC, quanTIseq, MCP-counter, ESTIMATE), signature scoring (PCA / z-score / ssGSEA / integration), TPM conversion and annotation, Immunophenoscore, ligand–receptor pairing, NMF and TME clustering, the full tme_profile chain, RNA-seq file merging, and the FASTQ→TME orchestration stages (fastp / salmon / STAR / TRUST4 / SpecHLA).

Omicos: the harness now exposes 27 typed analysis identifiers (the four signature modes are separate identifiers), including 16 new adapters. HLA and custom-reference BayesPrism remain direct-API capabilities. Input contracts and limits.

Evidence scope: R3–R6 timings below are imported campaign measurements. Some raw campaign scripts/logs are not archived here, so these numbers are not independently reproducible from this PR alone. New local validation separates numerical tests, stub-tool contracts and actual tutorial timings.

New real-data evidence: FASTQ/BAM/HLA recipe, repeated timings and logs and tutorials 24–28 use public sequencing reads and actual tools. Four-thread Salmon varies between repeated original runs as well as across wrappers; HLA extraction was slower in this small fixture. These results do not support a claim that every workflow is faster or universally byte-identical.

What's new in 0.3.0

• 18 additional workflow APIs (campaign rounds R3–R6): nmf, merge_salmon, merge_star_count, prepare_salmon, log2_eset, ips, mouse2human, lr_cal, tme_cluster, bayesprism, tme_profile, fastq_qc, batch_salmon, batch_star_count, trust4, runall, spechla, hla_typing, plus a 30.3× glue optimization inside calculate_sig_score.
• Standalone HLA read extraction: extract_hla_read extracts reads from one BAM/CRAM using prepared tools, with dependency auto-installation off by default.
• 4 new Rust kernels: lr_gene_valid_mask (LR_cal gene filter), tme_kmeans (k-means + KL index), merge_salmon_parse (quant.sf reader), bp_gibbs (BayesPrism Gibbs sampler reproducing numpy's full RNG chain bit-for-bit).
• tme_profile end-to-end 10.59× vs the original CLI on the frozen STAD fixture — the endpoint of a measured bottleneck-shift chain 1.12× → 8.44× → 10.59× (BENCHMARKS.md Part II).
• Campaign-reported benchmark (R6): 14 core candidates re-measured under one unified cold-start protocol against the unmodified original CLI; every parity contract reported PASS. The original submission recorded 188 default tests; current checks and limitations are tracked in validation.

Additional executed tutorials

Eleven new notebooks cover IPS, LR scores, NMF, TME clustering, conditional log transforms, mouse mapping, Salmon/STAR merging, Salmon preparation, tme_profile and BayesPrism. Each includes actual outputs and an embedded figure with two reviewed revisions. The table reports one final notebook API call on the local WSL/Omicos interpreter, two requested threads, excluding input preparation. These are small tutorial observations, not speedup claims. NMF's BLAS parallelism is not governed solely by the requested thread count. The BayesPrism example uses a short demo chain; file-merging examples are synthetic.

Install

Validated target: Python 3.11, Linux x86-64 / WSL2. Use an isolated environment. Versioned wheels, source archives, checksums and the container digest are delivered through the v0.3.1 release. After downloading its CPython 3.11 wheel:

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install --only-binary=:all: ./iobrx-0.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
python -c "import iobrx; print(iobrx.backend_info())"

This wheel path does not require Cargo or a C++ compiler. When 0.3.1 appears on PyPI, the index-based equivalent is:

python -m pip install --only-binary=:all: iobrx==0.3.1
# Tsinghua mirrors PyPI asynchronously; wait until the same version is listed.
python -m pip install --only-binary=:all: -i https://pypi.tuna.tsinghua.edu.cn/simple iobrx==0.3.1

PyPI publication and mirror synchronization are independent of GitHub assets. If an index has not synchronized, install the release wheel directly.

For source development, install Cargo and a C++17 compiler:

git clone https://github.com/LCGaoZzz/iobrx.git
cd iobrx
python -m pip install -c tests/constraints-validated.txt ".[tutorials,test]"
python -c "import iobrx; print(iobrx.backend_info())"
python -m jupyterlab tutorials

Use the same interpreter as the Jupyter kernel. In an existing Omicos environment, first check dependency compatibility; use an isolated environment when the required numerical versions conflict. pip install from this source builds the extension; it is not an installation without compilation.

The core-analysis container uses ghcr.io/lcgaozzz/iobrx:0.3.1; for immutable execution use the digest in the release's container-digest.txt. Alignment and HLA binaries require the separate external-tool environment. See release notes.

Why the dependency pins (details and measurements in BENCHMARKS.md §I.7):

These are the exact requirements in package metadata, not minimum versions. Remaining dependency bounds and the complete test environment are recorded in the validation constraints.

CPU compatibility and operating-system packaging are separate. Upstream IOBRpy's binary distribution limits straightforward installation on other Python/platform combinations. Windows users can use WSL2; macOS, ARM and native Windows are not claimed as validated full-stack targets. See the compatibility notes for fallback behavior and limits.

Quick start

import numpy as np
import pandas as pd
import iobrx

iobrx.set_threads(8)
counts = pd.read_parquet("tutorials/data/eset_stad.parquet")  # genes × samples
tpm = iobrx.count2tpm(counts, check_data=True, remove_version=True)
log_expression = np.log2(tpm + 1)

# Deconvolution and scores (linear TPM for the RNA-seq deconvolution examples)
cib = iobrx.cibersort(tpm, perm=100, QN=False)          # Rust NuSVR core
epic = iobrx.epic(tpm)["cellFractions"]
qnt = iobrx.quantiseq(tpm, tumor=True, rmgenes="default")
scores = iobrx.calculate_sig_score(
    log_expression, "signature_collection", method="integration"
)

# 0.3.0 ports
lr = iobrx.lr_cal(eset="tpm_symbol.csv", output_file="lr.csv",
                  data_type="tpm", id_type="symbol", cancer_type="pancan")
ips = iobrx.ips(eset="tpm_symbol.csv", output_file="ips.csv")
clusters = iobrx.tme_cluster(df=pd.read_csv("tme_transposed.csv"), id="sample")
iobrx.bayesprism(bulk="bulk_counts.csv", out_dir="bp_out", n_threads=8)
#   add backend="rust" for the bit-exact Rust Gibbs kernel

# The whole TME profiling chain in one process (10.59× vs the original CLI)
iobrx.tme_profile(input="TPM.csv", output="tme_out", threads=16)

# FASTQ → TME orchestration (external tools must be on PATH — see below)
iobrx.runall(mode="salmon", outdir="run_out", fastq="raw_fastq_dir",
             threads=16, resume=True,
             unknown=["--index", "references/salmon"])

runall(resume=True) reuses a table only after its producing step succeeded and its recorded output hash matches. A failed or interrupted table write is retried; successful upstream steps can still be reused. State schema 2 adds these per-step records: pre-schema-2 run directories require a new outdir. Existing schema 2 records remain compatible. Changed inputs, parameters, references or recorded calculation products require a new directory; unrelated notes and figures can be added, edited or removed without blocking resume. An uncatchable kill can leave unverified calculation products, which are not treated as completed work.

Open the complete workflow notebook for input checks, interpretation, plots, and per-stage timers on the public example data (no download needed after installation). The four standalone signature tutorials use the public IMvigor210 demonstration panel: 872 features × 348 samples.

The 29 APIs at a glance

Three routes, one choice — how every API is accelerated

Each ported module was evaluated along three routes: keep calling the original iobrpy CLI, write a pure-Python fast path, or build a Rust kernel. The final choice is whatever passed the bit-exact contract at the best measured wall time. Speedups for the 19 ported APIs are the R6 formal blind benchmark (unified cold-start protocol, fresh subprocess per run, interleaved arms, medians — research/bench_r6/results.json, BENCHMARKS.md §II.6); orchestration-stage ratios are the R5 real-data runs (§II.8); pre-port modules carry their official-gate numbers (Part I).

Why the speedups differ: one floor model

Every number in the table above obeys speedup ≈ min(1/(1−p), floor) — p is the share of the original wall time that is a compressible hotspot, and each module class has its own incompressible floor. This was the campaign's measured conclusion across rounds R1–R6 (BENCHMARKS.md Part II), not a post-hoc story:

1. A Rust kernel pays off (≥5×) only when p ≥ 0.6. lr_cal's per-gene pandas filter passes were p=0.81 of the original wall → 8.48×; tme_cluster's pure-Python k-means dominates → 7.17×. BayesPrism's Gibbs sampler is only p≈0.24 → the Rust kernel lands at 3.74× (1.56× over the python lane), below 5× despite being bit-exact by construction.
2. Bottleneck shift (multi-stage Amdahl). tme_profile: 1.12× (one process, original code — the un-optimized sig-score glue was ~84% of the wall) → 8.44× (the 30.3× glue fix moves the floor to cibersort + LR_cal ≈76%) → 10.59× (LR_cal→Rust, sub-step 4.10 s → 0.22 s ≈18.6×; the floor is now the contract-bound original CIBERSORT solver at 58.6% + the sig Rust chain at 31%). The next floor is ~9.3 s (~16.6×) — ≥12× is unreachable without breaking the cibersort bit-exact contract.
3. Write-out ceiling (~2.4×) for the io-merge modules. merge_salmon's Rust parser is 10.32× faster in isolation, but the byte-level output contract (pandas to_csv serialization + gzip) caps end-to-end at ~2.5×: measured 2.41–2.48× ≈ 96% of the same-window theoretical ceiling. Rust cannot buy its way past the write floor.
4. Import + pandas-IO floor (~2–3.8×) for the small modules. Every original CLI call pays a ~1.3–1.4 s iobrpy.main import on 1.3–1.9 s total walls. Removing it exposes pandas read_csv/to_csv as the new floor: pure-IO modules land at 2.29–2.37× (log2_eset, prepare_salmon), modules with real net compute at 3.25–3.76× (ips, mouse2human), and nmf converges to 1.21× because both arms run the same sklearn solver (shared-kernel floor).
5. Orchestration ceiling (speedup ≈ 1). For fastq_qc / batch_salmon / batch_star_count / trust4 / runall / spechla / hla_typing the external binaries are the wall. Real-data ratios: 0.99–1.25× (the 1.25× STAR residual is fully attributable to a declared 32→16 thread deviation, visible in STAR's own mapping-speed log). The ported value is API consistency, in-process composition (no console-script/PATH requirement), resume/parallel scheduling and byte-identical products — plus a Python launch layer 3.3–14.9× lighter, measurable when the tools are stubbed.

Bit-exact contracts and known nondeterminism

Bit-exact means: identical index/columns/dtypes/NaN mask with max_abs_diff == 0.0 on every numeric cell, or sha256-equal output files, versus the ORIGINAL iobrpy 0.2.0 executed in the same environment. The contracts are enforced by tests/test_parity_*.py (188 tests pass by default; the official-data gates run in CI with the frozen fixtures).

Every known nondeterministic item is an upstream property, proven by original-vs-original controls, and handled by an explicit contract clause:

• CIBERSORT P-value column — the original seeds its permutations from OS entropy (SeedSequence() unseeded), so the original itself is not reproducible run-to-run on that column. iobrx seeds them: stable across runs and thread counts, identical formula and 1/perm granularity. The column is excluded from every byte contract that contains it (cibersort output; tme_profile's cibersort_results.csv / deconvo_merged.csv).
• BayesPrism state order — the original's cell-state iteration order follows the per-process PYTHONHASHSEED and can flip discrete Gibbs draws. iobrx's default state_order="sorted" removes the dependency (theta / theta_cv 100% bit-exact vs a PYTHONHASHSEED=0 original; Z_tumor ULP-only, max abs 1.42e-14, zero discrete flips); state_order="legacy" with PYTHONHASHSEED=0 reproduces the frozen gold shas exactly (the R6 parity configuration).
• as_completed column order — the original merge_salmon / merge_star_count column order varies run-to-run (three different orders observed in three R6 reps). iobrx emits a deterministic sorted order; the parity contract aligns by column name and compares token-for-token and frame-bit-for-bit.
• samtools @PG / @RG header IDs — spechla / hla_typing <sample>.realign.sort.bam inherit random samtools merge header ids (e.g. bwa-7A10F178); an original-vs-original rerun shows the same jitter. Alignment record streams and every other product are byte-compared.
• fastp HTML jitter — the duplication rate's 7th significant digit in fastp.html varies per run inside the fastp binary (an original CLI rerun reproduced the port's digit and differed from its own frozen baseline); cleaned FASTQs are byte-identical.
• Run metadata — log timestamps (salmon/STAR/TRUST4 logs, multiqc uuid/creation dates) are declared run-metadata; every data artifact is sha256-compared without normalization.
• Bug-compat preserved — upstream defects that are visible in output bytes are kept deliberately, e.g. merge_star_count's leading four global stat rows that the original never purges, and lr_cal's count-branch behavior.

Exact parity is an observed result on specified inputs and pinned dependency versions (numpy<2.3, scikit-learn<1.8 — see Install), not a cross-platform floating-point guarantee (details).

External tools (bring your own)

iobrx orchestrates the heavy binaries but does not bundle them. Install them yourself and put them on PATH (or pass the per-stage *_bin overrides); iobrx preserves the upstream command parameters. Equality still depends on the tool's determinism and the declared comparison target; see the measured Salmon variability and scoped FASTQ/BAM/HLA comparisons in the new recipe.

All pure-compute APIs (deconvolution, signature scores, TPM, annotation, IPS, LR_cal, NMF/TME clustering, merges) need no external tools — only iobrpy (references + fallback) and the pinned scientific stack.

For extraction without typing, use the prepared environment:

iobrx.extract_hla_read("sample1", "sample1.bam", "hg38", "hla_reads")

This new API defaults to auto_install=False for both backends; explicitly enable auto_install=True only when installing tools is part of the task. It is currently a direct Python API, not a typed harness adapter.

Omicos and agent workflows

The agent harness adds a JSON CLI, an optional stdio MCP server and a portable Omicos Agent/Skill pair for 27 analysis identifiers. It validates matrix orientation, declared scale and gene IDs, then records parameters, input/output metadata, versions, backend and elapsed time in a result manifest. Analyses use the existing iobrx API and preserve its result layouts. Known requests can run directly; discovery and diagnostics are optional. Agents can also use the public Python API for custom work. Full SHA-256 auditing is opt-in; default result inspection checks file availability.

After installing iobrx, from this checkout:

python -m pip install ./agent-harness
iobrx-agent run --request agent-harness/examples/signature_pca.json

See Omicos setup for workspace/catalog installation and MCP configuration, and harness validation for the actual test record. The companion harness is installed from this repository; it is not yet a separately published PyPI package.

How long does each analysis take?

Measured on Intel Core i9-13900KF, WSL2 Ubuntu, Python 3.11, 8 requested threads, no AVX-512, using the existing Omicos environment. Each row runs in a fresh Python process: first call, followed by three repeat calls. The main time is the median of those three repeats. Data loading, input preparation and plotting are excluded; the complete workflow includes its own normalization and analysis stages. OS file caches may already be warm.

These are local wall-clock measurements, not a promise for other hardware or cohorts. CIBERSORT uses 100 permutations and QN=False on the full STAD TPM matrix. The restricted IMvigor210 signature panel is a different workload. The complete workflow runs integration scoring on the full STAD expression matrix, so its total is not the sum of the standalone rows.

The Parity column is not a timing: it states what the official gates (tests/test_parity_official.py) assert for that analysis on the same fixtures — equal index, labels and column names, and exact equality of every numeric cell (max_abs_diff == 0.0) against the ORIGINAL iobrpy implementations executed in the same environment (validation record). The single exception is CIBERSORT's P-value: the original seeds its permutations from OS entropy and is not reproducible run-to-run even by itself, whereas iobrx's P-values are seeded — stable across runs and thread counts, with the identical formula and 1/perm granularity. All 11 gates are re-run by CI on every push and pull request under the validated constraints, and the same contract held on the historical 224-thread Xeon campaign (BENCHMARKS.md). Exact parity is an observed result on those environments, not a cross-platform floating-point guarantee (details).

Raw repeats, ranges and environment · Benchmark method and reproduction · Historical Xeon performance campaign Full performance record: acceleration stack + port campaign

The historical 224-thread Xeon speedups (Part I) and the R6 blind port benchmark (Part II) are retained as separate experiments; they are not used to advertise desktop performance.

Choose the right output

The upstream method named zscore averages preprocessed signature expression; it does not guarantee standardized output. integration concatenates three methods rather than averaging them. Only the exact platform="affymetrix" string requests IOBRpy's calibrated ESTIMATE purity transformation; the historical "affy" default does not. These RNA-seq tutorials therefore show scores without that purity transformation.

Compatibility and numerical fidelity

iobrx.backend_info()  # native availability, sorting dispatch, bundled BLAS

# Optional: explicitly use the original IOBRpy workflow.
cib = iobrx.cibersort(tpm, perm=100, QN=False, backend="python")
# backend="rust" requires native support and reports an error if unavailable.

backend="auto" is the default wherever an accelerated lane exists (see the route table for which lane each module ships). Set IOBRX_DISABLE_RUST=1 before importing to disable native acceleration process-wide. Missing native support does not prevent using the public analysis API when IOBRpy and its dependencies are installed. Missing compatible OpenBLAS triggers fallback for CIBERSORT/PCA.

All parity gates run in CI on every push and pull request under the validated constraints; their assertion is exact equality — labels and every numeric cell, or file sha256 — against the ORIGINAL executed on the same fixtures, with the documented nondeterminism clauses above. Exact parity is an observed result on specified inputs and dependency versions, not a cross-platform floating-point guarantee. Native CIBERSORT uses seeded permutations; the original Python implementation uses unseeded permutations, so its P-value column is excluded from exact-equality assertions. See precision and fallback details.

Reproduce, test, contribute

python -m pytest -q                      # 188 smoke + parity + portability tests
IOBRX_TESTDATA=tutorials/data python -m pytest -q -m full   # official-data gates
python scripts/execute_tutorials.py      # all 12, fresh kernels, embedded plots
python scripts/benchmark_tutorials.py    # first call + 3 repeats per analysis
python scripts/validate_tutorials.py     # outputs, exports and data checksums

Tutorials can be read directly on GitHub. To run them interactively: python -m jupyterlab tutorials. Plotting helpers are in tutorials/_common.py; every analysis call remains visible in its notebook. The figure-review record documents both refinements, with draft and final overview images.

Contributions should preserve the parity gates and add a focused regression test for changed numerical behavior. Files in bench/ are frozen historical artifacts. Report package versions, backend information, input shape and expression scale when reporting a problem.

Upstream credit, citation and license

We thank the IOBRpy developers, the IOBR team, and the original method authors for the workflows and reference resources on which this package builds.

When publishing results, cite IOBR/IOBRpy and the original methods actually used — including CIBERSORT (Newman et al.), BayesPrism, quanTIseq, EPIC, MCP-counter, ESTIMATE, IPS (Charoentong et al. 2017), TRUST4, SpecHLA, and the tool papers for fastp/salmon/STAR when you run the orchestration stages. See the IOBRpy preprint and the upstream citation guide. The preprint reports CIBERSORT thread scaling inside the original implementation — a different measurement axis from the port speedups here (see BENCHMARKS.md §II.9).

iobrx provides acceleration and tutorials; it does not replace those methods or validate clinical conclusions from an unlabeled example dataset.

iobrx code: MIT. Vendored sources retain their third-party notices. Public example data retains its upstream attribution and GPL-3 terms.