colstore 0.2.0

Memory-mapped columnar binary format for fast random-access I/O on structured arrays.

ColStore

A memory-mapped columnar binary format for fast, memory-efficient I/O on structured arrays. colstore lets you write a tabular dataset to a single .cstore file (in one shot or streamed across many records) and then load arbitrary row/column subsets without materializing the rest. Internally, columns are stored back-to-back as raw NumPy bytes, reads use np.memmap, and fancy-index gathers run through a parallel C++ kernel (OpenMP + software prefetching) bound via Cython. Process memory stays bounded by the size of the output you ask for; the source file is never fully read into RAM.

Install

pip install colstore

Building from source needs a C++17 compiler and CMake ≥ 3.18. On macOS install libomp (brew install libomp) to get the parallel kernel; without it the build still succeeds but the kernel runs single-threaded.

Quick start

import colstore

# One-shot write + open. `.cstore` is the canonical extension.
ds = colstore.store(df, "data.cstore")

# Or open an existing file for read.
ds = colstore.open("data.cstore")

# Indexing returns lazy views; no data is read yet.
ds['price']                          # ColumnView
ds[100:200]                          # TableView
ds[100:200, 'price']                 # ColumnView
ds[100:200, ['price', 'qty']]        # TableView
ds[[1, 5, 9], ['price', 'qty']]      # TableView (fancy rows + cols)

# Materialize through one of the materialization methods.
ds['price'].array()                          # 1D ndarray
ds[indices, ['price', 'qty']].dict()         # dict of 1D arrays
ds[indices, ['price', 'qty']].recarray()     # structured ndarray
ds[indices, ['price', 'qty']].frame()        # pandas DataFrame

Writing

colstore.store(data, path) is the one-shot path; it dispatches on the input type:

import colstore
import numpy as np

# From a dict of 1D arrays.
colstore.store(
    {"x": np.arange(100, dtype=np.float32), "y": np.arange(100, dtype=np.int64)},
    "data.cstore",
)

# From a structured (record) array.
records = np.empty(100, dtype=[("price", np.float32), ("qty", np.int32)])
colstore.store(records, "data.cstore", mode="recreate")

# From a pandas DataFrame.
colstore.store(df, "data.cstore", mode="recreate")

mode="create" (default) refuses to overwrite; mode="recreate" truncates an existing file.

For multi-record streaming writes (data arriving in batches, or appending to an existing file), use colstore.create / colstore.recreate / colstore.update:

# Append-only stream into a new file.
with colstore.create("data.cstore") as f:
    for batch in source:
        f.write({"x": batch.x, "y": batch.y})

# Resume appending to an existing file. Schema is loaded from the manifest
# and every write must match it. Bytes from a crashed prior writer are
# truncated on open.
with colstore.update("data.cstore") as f:
    f.write({"x": more_x, "y": more_y})

The writer commits records atomically on close() by rewriting a 32-byte counters block. A reader opening the file mid-write sees only what the last successful close committed.

Compaction

A streaming write produces one record per write() call. Reads of slice and sorted-fancy index patterns scale near-flat with record count, but unsorted-fancy reads degrade as records accumulate. colstore.compact collapses all records into one, so every read pattern takes the single-record fast path:

colstore.compact("data.cstore")                 # in-place
colstore.compact("data.cstore", out="new.cstore")  # leave source untouched

In-place compaction writes to a sibling temp file and atomically renames into place; the source is untouched on failure. The byte copy uses os.sendfile on Linux (kernel-space copy, no Python-side surfacing) and shutil.copyfileobj on macOS/Windows; on both paths memory footprint is bounded by the kernel/I/O buffer (tens of KB) regardless of file size — files much larger than RAM compact fine.

Introspection

i = colstore.info("data.cstore")
# ColStoreInfo(path='data.cstore', n_rows=1_000_000, n_records=42,
#              columns=[a:<f4, b:<i8], file_size=8_001_232B,
#              needs_compaction=True)

colstore.schema("data.cstore")
# [{'name': 'a', 'dtype': '<f4', 'encoding': 'raw', 'nullable': False}, ...]

Both info and schema read only the file header (no record bodies are scanned), so they're cheap on multi-GB files.

Configuration

from colstore import set_max_workers, set_default_madvise, set_default_backend

set_max_workers(8)                # parallel gathers across columns
set_default_madvise("sequential") # OS read-ahead hint for sorted-index reads
set_default_backend("cpp")        # gather kernel: cpp | numpy | numba

On-disk format

[magic 8B = b"CSTORE\x00\x01"]
[counters 32B: n_records(8) + committed_rows(8) + crc32(4) + reserved(12)]
[manifest_len 8B (u64 little-endian)]
[manifest_json: format_version + columns + manifest_crc32]
[zero-padding to 64-byte alignment]
[record_0 header 32B][record_0 body, padded to 8B]
[record_1 header 32B][record_1 body, padded to 8B]
...

The JSON manifest is immutable and carries only the schema; the mutable record/row counters live in their own fixed-position 32-byte block (with its own CRC) right after the magic. Each record body holds the columns back-to-back as raw bytes. A one-shot write produces a single-record file that reads via a per-column memmap fast path; a streamed write produces a multi-record file with per-pattern dispatch (contiguous range, sorted fancy, unsorted fancy).

Supported dtypes

All fixed-size NumPy dtypes are supported: float32/float64, int8/16/32/64, uint8/16/32/64, bool, datetime64, timedelta64, and fixed-width strings (S bytes and U unicode, with the width baked into the dtype, e.g. S16 or U8). Object dtype (variable-length strings, Python objects) is rejected at write time — the design point is zero-copy random access, which requires a fixed stride per row.

License

MIT License - see LICENSE file for details.