npcomp — lossless compression and backup for SpikeGLX Neuropixels data

npcomp is a minimal CLI wrapper around mtscomp, the IBL lossless compressor for raw Neuropixels data (about 3x compression):

• compress: losslessly compress SpikeGLX .bin files into .cbin + .ch (mtscomp's built-in byte-per-byte verification runs automatically);
• restore: decompress .cbin + .ch back to .bin (also automatically verified); --dest mirrors the whole backup back;
• status: report compression state, space already saved, estimated savings and time;
• backup mode (--dest): mirror a whole folder into a destination; .bin files are compressed, everything else (including empty directories) is copied and verified.

Chinese version: README_ZH.md.

Installation

From PyPI:

pip install npcomp

From source (current development workflow):

conda create -n mtscomp python=3.10 -y
conda activate mtscomp
cd <path-to-repo>
pip install -e .

Add the Scripts folder of the conda environment (e.g. <conda-env>\Scripts, where npcomp.exe lives) to your user PATH, then open a new terminal and run npcomp directly.

Quick start

# Report compression status
npcomp status <path-to-session>

# Compress in place (.bin -> .cbin + .ch, originals kept)
npcomp compress <path-to-session>

# Mirror backup to a destination: .bin compressed, everything else copied & verified
npcomp compress <path-to-session> --dest <backup-dir>

# Same backup, but run .bin compression through a fast scratch disk (e.g. NVMe)
npcomp compress <path-to-session> --dest <backup-dir> --scratch <fast-disk-dir>

# In-place compression through a fast scratch disk (no --dest needed)
npcomp compress <path-to-session> --scratch <fast-disk-dir>

# Delete source .bin once it is compressed and verified (existing verified pairs too)
npcomp compress <path-to-session> --dest <backup-dir> --delete

# Restore (in place by default; --dest mirrors the backup back)
npcomp restore <backup-dir>
npcomp restore <backup-dir> --dest <restore-dir>

If your raw .bin files are on a mechanical HDD, strongly recommend setting --scratch to an SSD/NVMe. mtscomp reads chunks with many parallel threads - a random-access pattern mechanical disks handle poorly - while --scratch turns the HDD read into a single sequential copy. In our tests, runs were about 3x slower without it.

Command reference

npcomp compress <folder> [--dest DIR] [--scratch DIR] [--dry-run] [--delete] [--cpus N] [--chunk SEC] [--force] [--yes] [--quiet]

• Recursively scans <folder> and only processes .bin files that have a valid SpikeGLX .meta sibling; covers ap/lf/nidq and leaves every other .bin untouched.
• Parameters (channel count, sample rate) are read from the .meta; the data type is fixed to int16.
• Defaults: 1-second chunks, all CPU cores, verification always on.
• Without --dest: writes X.cbin + X.ch next to the source and records their hashes in .npcomp_manifest.json (fast verified --delete); verified existing pairs are skipped, mismatched pairs are detected and recompressed after confirmation, --force redoes everything.
• With --dest: mirror-backup mode — .bin files are compressed into the destination, everything else is copied and byte-verified (directory structure included). Identical existing files are skipped; differing ones prompt (--yes accepts all, --force redoes everything).
• With --scratch: raw .bin files are first copied to a fast scratch directory (e.g. an NVMe SSD), compressed there, and the verified .cbin/.ch are copied back to the destination (or written next to the source when no --dest is given). Recommended when the source .bin files are on a mechanical HDD or the destination is a slow disk (see the note above). Scratch files are removed after each file; the scratch directory must be empty before a run (npcomp refuses otherwise) and should stay dedicated to npcomp.
• --delete: delete the source .bin for every recording whose compressed pair is verified - compressed in this run, or an existing pair confirmed to match the source. Unverified or conflicting pairs keep the source.
• --dry-run: print the plan (compress/copy/delete) without touching anything.
• If the destination already contains data (e.g. a previous run interrupted by power loss), you are prompted first; non-interactive runs require --yes.

npcomp restore <folder> [--dest DIR] [--overwrite] [--delete] [--dry-run] [--quiet]

• Recursively finds every X.cbin + X.ch pair and decompresses it to X.bin (mtscomp verifies each decompression byte-for-byte).
• Restores in place by default; existing .bin files are skipped unless --overwrite is given.
• With --dest: full mirror restore — pairs are decompressed and everything else is copied back byte-verified (directory structure included). Existing files are kept; --overwrite replaces differing ones.
• The compressed X.cbin + X.ch pair is kept by default; only an explicit --delete removes it after a successful restore (and only pairs successfully restored in this run; skipped pairs are kept).
• A missing .meta only produces a warning — it is not needed for byte-level decompression, but downstream tools need it.

npcomp status <folder> [--ratio F] [--throughput MBS]

• Reports per-file state (raw-only / both / compressed-only / incomplete, plus +tmp for leftovers) and sizes.
• Compressed files use actual sizes for the saved-space figure; uncompressed files are estimated with the default compression ratio 3.0 (--ratio) and throughput 88 MB/s (--throughput).
• Status is based on file existence and size only; it does not validate file contents.

Safety design

• Lossless + automatic verification: compression and decompression both run a byte-per-byte check; failures raise errors and clean up temp files, so no half-written output survives.
• Atomic rename: data is written to *.tmp and renamed into place only after verification; leftover .tmp files are cleaned automatically on the next run (only npcomp's own files — unrelated tmp files are never touched). Source .bin files are never touched by a failed run.
• Conservative deletion: only an explicit --delete removes source .bin files, and only when their compressed pair is verified (compressed in this run, or an existing pair confirmed byte-identical to the source).
• Backup verification: in --dest mode every copied file is byte-verified after copying; .npcomp_manifest.json in the destination records the SHA-256 of every source file (plus the .cbin/.ch hashes for recordings), so re-runs skip unchanged content and detect same-name/different-content or tampered files. In-place compression writes the same manifest next to the source, so repeated --delete runs confirm pairs by hash without re-decompressing.
• Restore verification: restore --dest copies non-recording files back with the same byte verification, so the restored tree mirrors the backup.
• Space check before writing: compress estimates the output size (raw size / 3 plus a small per-file overhead) and refuses to start when the target drive — or the scratch drive, against the largest single file — lacks room. If a disk still fills up mid-run, the run aborts immediately instead of failing file by file.
• Two-step rename caveat: .cbin and .ch are each renamed into place in separate steps; a power loss in between can leave a mismatched pair. The next compress run detects this via the manifest and prompts to recompress (self-healing).
• Keep the trio: after backup, keep X.cbin + X.ch + X.meta together. Losing .ch makes decompression impossible; losing .meta makes the restored data unusable by downstream tools.

License

This project is licensed under the BSD 3-Clause License.