BCL Convert Pipeline

A toolkit for Illumina BCL-Convert sample-sheet generation, demultiplexing, and 10x single-cell Excel-to-samplesheet transforms.

The unified bclconvert tool

The canonical, standalone tool lives in src/bclconvert/ — a single installable package with a bclconvert CLI that unifies sample-sheet generation, demultiplexing, and the single-cell Excel transform. It is polars-based and was consolidated from bcl2airr plus the legacy scripts in this repo. bcl2airr depends on it via thin re-export shims (see docs/bcl2airr-shim-migration.md).

pip install -e .            # core (polars + click)
pip install -e '.[excel]'   # + pandas/openpyxl for the `transform` command

CLI reference

All commands support a global -v/--verbose flag (must come before the subcommand, e.g. bclconvert -v make-sheet ...) for debug logging.

make-sheet — generate a SampleSheet v2

# All-barcodes mode: emit every barcode in the chosen kit(s) — no sample list
bclconvert make-sheet --run-dir RUN/ --barcode-type dual --kit all -o SampleSheet.csv

# Known-samples mode: look up i7/i5 for a Lane,Sample_ID,Index list
bclconvert make-sheet --run-dir RUN/ --samples samples.csv \
    --barcode-type dual --kit TT,TN -o SampleSheet.csv

# Custom (non-bundled) index file, forward-strand i5
bclconvert make-sheet --barcode-type dual --index-file my_index.csv \
    --i5-workflow A --runinfo RUN/RunInfo.xml -o SampleSheet.csv

Option	Description
--barcode-type {dual,single}	Required. Barcode kit type used for the run.
--kit CODES	Bundled kit code(s), comma-separated (TT,TN) or all. Defaults to all unless --index-file is given.
--index-file PATH	Custom Illumina-format index CSV (repeatable). Used in addition to --kit.
--i5-workflow {A,B}	i5 orientation. Default B (reverse complement — correct for NextSeq 550/1k/2k and NovaSeq 6000). A is forward strand.
--samples PATH	Known-samples CSV (Lane,Sample_ID,Index, or Sample_ID,Index). Omit for all-barcodes mode. Lane may be an integer or * (bcl-convert's native "all lanes" wildcard), and is passed through unchanged.
--run-dir PATH	Run folder containing RunInfo.xml and/or RunParameters.xml.
--runinfo PATH	Explicit RunInfo.xml path (overrides --run-dir).
--runparameters PATH	Explicit RunParameters.xml path (overrides --run-dir).
--run-name NAME	Override the RunName in [Header].
-o, --output PATH	Required. Output SampleSheet.csv path.

run — execute bcl-convert

bclconvert run --bcl-dir RUN/ --sample-sheet SampleSheet.csv --output-dir out/ --threads 8

--dry-run prints the bcl-convert command without executing it.

pipeline — make-sheet + run in one step

bclconvert pipeline --bcl-dir RUN/ --barcode-type dual --kit all --output-dir out/

Takes the same barcode/kit/i5 options as make-sheet, plus --sample-sheet-name (default SampleSheet.csv), --threads, and --dry-run.

transform — single-cell Excel → sample CSV

Needs the [excel] extra (pip install -e '.[excel]').

bclconvert transform -e plate.xlsx -f /path/to/fastq -o samples.csv

validate — check a SampleSheet v2

bclconvert validate SampleSheet.csv

Exits non-zero and prints INVALID: <path> if required sections or data rows are missing; prints OK: <path> otherwise.

File formats

Custom index CSV (--index-file)

Dual-index kits use the Illumina 10x format:

index_name,index(i7),index2_workflow_a(i5),index2_workflow_b(i5)
SI-TT-A1,GTAACATGCG,AGTGTTACCT,AGGTAACACT

Known-samples CSV (--samples)

Lane,Sample_ID,Index
1,Sample1,SI-TT-A1
*,Sample2,SI-TT-A2

Sample_ID/Index (case-insensitive) are matched by header name; Lane is optional. Headerless files fall back to positional [Lane,] Sample_ID, Index order. Index must match an index_name from the loaded kit(s)/index file(s).

Output SampleSheet v2 structure

[Header]
FileFormatVersion,2
RunName,<run_id>

[Reads]
Read1Cycles,<n>
Read2Cycles,<n>

[BCLConvert_Settings]
CreateFastqForIndexReads,1
MinimumTrimmedReadLength,8
FastqCompressionFormat,gzip
MaskShortReads,8

[BCLConvert_Data]
Lane,Sample_ID,Index,Index2   # Lane column omitted in all-barcodes mode
...

Python API

Everything the CLI does is backed by a public library API exported from bclconvert/__init__.py — not just internal glue for cli.py. pip install -e . gets you both.

from pathlib import Path
import bclconvert

# Load barcodes for bundled kit(s) (polars DataFrame: index_name, i7, i5)
barcodes = bclconvert.load_index("dual", kits=["TT", "TN"])

# Resolve run metadata from RunInfo.xml/RunParameters.xml
meta = bclconvert.resolve_run_metadata(run_dir="RUN/")

# Build a SampleSheetV2 in-memory (all-barcodes or known-samples mode)
sheet = bclconvert.build_samplesheet(barcodes, run_meta=meta, samples="samples.csv")
sheet.sample_count      # int
sheet.to_csv()          # str, the full SampleSheet.csv content
sheet.write("out.csv")  # str or Path

# Parse an existing sample sheet back into the same model.
# parse_samplesheet_v2 reads from disk given a Path; a str is parsed as literal
# CSV text. Passing a path as a str raises ValueError rather than failing quietly.
parsed = bclconvert.parse_samplesheet_v2(Path("SampleSheet.csv"))
parsed = bclconvert.parse_samplesheet_v2(sheet.to_csv())  # str = CSV text
bclconvert.validate_sample_sheet("SampleSheet.csv")  # bool; str or Path both fine here

# Run bcl-convert itself (thin subprocess wrapper, returns exit code)
bclconvert.run_bcl_convert(bcl_dir="RUN/", sample_sheet="SampleSheet.csv", threads=8)

Exported surface (bclconvert.__all__): kit constants (AVAILABLE_KITS, KIT_CODES, DUAL_INDEX_KITS, SINGLE_INDEX_KITS, NORMALIZED_COLUMNS), load_barcode_kit/load_index/load_all_dual_index_barcodes/resolve_kit_names, RunMetadata/parse_runinfo/parse_runparameters/resolve_run_metadata, build_samplesheet, SampleSheetV2/parse_samplesheet_v2/validate_sample_sheet, barcodes_to_samples/samples_to_barcodes, run_bcl_convert. This is the same surface bcl2airr consumes via thin re-export shims (see docs/bcl2airr-shim-migration.md), so it's designed for cross-package use, not CLI-only.

Legacy scripts — verification baseline

The original pandas scripts, retained for cross-checking the unified tool's output during the verification window, now live under src/bclconvert/legacy/ — see that directory's README for usage.

Repository structure

bcl-convert/
├── src/bclconvert/             # Canonical, unified package + CLI
│   ├── cli.py                  #   bclconvert CLI: make-sheet/run/pipeline/transform/validate
│   ├── builder.py              #   build SampleSheetV2 (all-barcodes + known-samples)
│   ├── runinfo.py              #   RunInfo.xml / RunParameters.xml metadata parsing
│   ├── runner.py                #   bcl-convert execution wrapper
│   ├── transform.py            #   single-cell Excel → sample CSV (HTO→CMO)
│   ├── samplesheet.py          #   SampleSheetV2 model + v2 parser/writer
│   ├── legacy/                 #   Retained pandas scripts (verification baseline; see legacy/README.md)
│   └── barcodes/
│       ├── kits.py             #   10x barcode-kit loaders + kit selection (polars)
│       └── data/*.csv          #   5 bundled 10x kit CSVs
├── pyproject.toml              # Packaging; entry point: bclconvert = cli:main
├── docs/
│   └── bcl2airr-shim-migration.md  # Plan to repoint bcl2airr at bclconvert
├── tests/                      # unit/integration/performance test suites
├── Index/                      # Legacy index files (subset, duplicate of barcodes/data)
└── README.md                   # This documentation

Requirements

• Python >= 3.9
• click, polars (core)
• pandas, openpyxl (only for transform, via the [excel] extra)
• bcl-convert (Illumina, for the run/pipeline commands)

Troubleshooting

1. Missing bcl-convert: ensure the Illumina bcl-convert binary is installed and on PATH — bclconvert run/pipeline shell out to it.
2. FileNotFoundError on kit loading: check --kit codes match bundled kits (TT, TN, TS dual; N, T single) or that --index-file points at a valid Illumina-format CSV.
3. ... index name(s) not found in the loaded barcode kit(s): the --samples CSV references an Index name not present in the kit(s)/index file(s) you loaded — add the missing kit via --kit.
4. Dry run: add --dry-run to run/pipeline to print the bcl-convert command without executing it.