blobpack

Pack your dataset's media into dumb zip shards. Read them fast anywhere.

A Blob Pack is a directory of uncompressed (STORED) zip shards plus a convention for referencing their members from annotation tables. It is not a new format: any zip tool can read it, no library is required to consume a dataset built with it, and your media bytes are stored unchanged.

Install and quickstart

pip install blobpack

No dependencies in the core. Python >= 3.9.

import os

from blobpack import PackWriter, PackSet

# write: one directory of rolling shards (4 GiB files by default)
with PackWriter("media") as writer:
    refs = [writer.add(f"images/{i:04d}.jpg", os.urandom(1024)) for i in range(1000)]
    # refs[0] == "zip://images/0000.jpg::media/pack-0000.zip"

# read: opens each shard's index once, then one positioned read per blob
with PackSet("media") as packs:
    data = packs.read(refs[0])  # or packs.read("images/0000.jpg")
    batch = packs.read_many(refs, workers=8)  # concurrent, input order

    # big blobs: a bounded seekable file object, nothing loaded up front
    with packs.open(refs[0]) as handle:
        header = handle.read(2)  # hand to a decoder; seeks stay inside this blob

    # epoch pattern: shuffled shards, sequential within each shard,
    # split across dataloader workers by member range
    for key, data in packs.iter_blobs(shuffle_shards=True, seed=0, worker_id=0, num_workers=4):
        ...

writer.add_file(key, path) streams an existing file in without holding it in memory. From the shell:

blobpack pack   ./images ./media      # files -> shards
blobpack verify ./media               # STORED + unique keys + full CRC pass
blobpack ls     ./media
blobpack unpack ./media ./restored

zip://images/0000.jpg::media/pack-0000.zip
      ^ member name       ^ shard path, relative to the dataset root

unzip -p media/pack-0000.zip images/0000.jpg > out.jpg

One shape for images, audio, and robot episodes

Every modality hits the same wall: millions of small files are miserable to store, copy, and read, while burying the bytes inside a table makes every random read expensive. Blob Pack separates the two concerns — tables keep the annotations, packs keep the payloads:

The mechanism is the same everywhere: a loose-file read costs per file (milliseconds of metadata, whatever the size — the exact cost varies with filesystem and layout), while a pack read costs per byte. The gap is therefore widest where files are small and many:

Compared with the alternatives

Six things a dataset team ends up needing, in the regime blobpack targets (many media blobs, shared or object storage) — starting with holding millions of blobs as tens of files. ✅ out of the box · ⚠️ with extra machinery or non-default settings · ❌ not available:

_{¹ with a sidecar index (e.g. wids) · ² the WebDataset idiom keeps annotations inside shards, next to each sample · ³ payloads are only reachable through the Lance engine, and the table itself becomes Lance · ⁴ row-group granularity; shrinking row groups to approximate random access trades away parquet's own scan efficiency}

Every alternative either gives something up or needs extra machinery to get it back; blobpack needs neither, in exchange for one hard scope line: it holds immutable blobs and lets anything address them, full stop. Tables, queries, and mutation stay in whatever you already use — Hugging Face Datasets, parquet, or Lance when you need column queries and updates. Skip blobpack when data mutates in place, when a pure byte-pipe stream with no seeks is all you need (tar's one exclusive trick), or for millions of tiny scalars, which belong in the table.

Performance

Measured end to end through a torch DataLoader with persistent workers decoding every JPEG (COCO train2017, 40,000 images):

On a shared cluster filesystem, blobpack sustains 2.5x loose files on random access and edges past WebDataset when streaming — while keeping random access, which streaming formats give up. On local NVMe every format converges because JPEG decoding becomes the bottleneck; there the isolated storage layer serves blobs at 0.035 ms each, 21.5x faster than stock zipfile on the same shards. Scaling with workers is clean:

Startup is a one-time cost, and the catalog removes it: a 40,000-member set opens in 0.35 s instead of 12 s. Full methodology, audio and video results, startup and shard-size sweeps, and every caveat: benchmarks/.

With Hugging Face Datasets

References are plain strings, so tables need nothing special:

from datasets import Dataset, load_from_disk
from blobpack import PackWriter, PackSet

with PackWriter("dataset/media") as writer:
    refs = [writer.add(f"img/{i:06d}.jpg", img) for i, img in enumerate(images)]
Dataset.from_dict({"image_ref": refs, "label": labels}).save_to_disk("dataset/table")

# ... later, anywhere the folder was copied to:
ds = load_from_disk("dataset/table")
packs = PackSet("dataset/media")
img = packs.read(ds[0]["image_ref"])

Everything under dataset/ is relative, so the folder is self-contained: move it across machines or clusters and it keeps working.

Storage and distribution

Very large pack sets

Opening a pack set parses every shard's index — fine for thousands of blobs, wasteful for millions. catalog=True keeps the mapping in a SQLite file next to the shards, so later opens cost a few queries:

packs = PackSet("media", catalog=True)  # builds once, reuses afterwards

The catalog is a pure cache: it is reused only while every shard keeps the exact identity recorded at build time, and anything else rebuilds it with full validation — a shard swapped in place is never served through stale offsets.

Object storage

Packs work unchanged on anything fsspec speaks: the index builds from batched concurrent range reads, each blob is one ranged request, and a backend that ignores byte ranges is refused rather than silently downloading whole shards. Protocols need their backend package (pip install "blobpack[remote]" s3fs):

packs = PackSet("s3://bucket/dataset/media", storage_options={"anon": False})
data = packs.read("images/0000.jpg")
# or bring your own configured filesystem: PackSet.from_fs(fs, "bucket/dataset/media")

Publishing on the Hugging Face Hub

A pack dataset is an ordinary folder: publish with upload_folder, and consumers stream blobs straight off the Hub or download as usual (pip install "blobpack[remote]" huggingface_hub):

from huggingface_hub import HfApi
from blobpack import PackSet

REPO = "your-name/your-dataset"
api = HfApi()
api.create_repo(REPO, repo_type="dataset", exist_ok=True)
api.upload_folder(repo_id=REPO, repo_type="dataset", folder_path="dataset")
packs = PackSet(f"hf://datasets/{REPO}/media")  # no download; ranged reads

examples/hf_hub.py runs the whole loop, and a live three-domain demo — PASS images, LibriSpeech utterances, and PushT episodes cut into per-episode containers by convert-lerobot — is up at MilkClouds/blobpack-demo.

Migrating from other formats

Existing datasets convert in one command, and nothing is ever re-encoded: media bytes come out of a conversion exactly as they went into the source. Three layouts are supported, each behind its own extra (pip install "blobpack[lerobot]" / "blobpack[lance]"; WebDataset needs none):

WebDataset conversion changes nothing but the container, so it just runs. The other two can rewrite table schemas, so they print a plan of exactly what each feature or column will become and wait for your confirmation (--dry-run stops at the plan, --yes skips the prompt). Per-source detail folds out below.

blobpack convert-wds ./shards ./media                  # *.tar -> packs
blobpack convert-wds ./shards ./media --shard-prefix   # if keys repeat across tars

before                              after
shards/                             media/
  shard-0000.tar                      pack-0000.zip
    000000.jpg  000000.json             000000.jpg  000000.json
    000001.jpg  000001.json             000001.jpg  000001.json
  shard-0001.tar                      pack-0001.zip
    000002.jpg  000002.json             000002.jpg  000002.json

sequential only, no member index    same bytes, plus a member index:
                                    random reads, sub-shard worker splits

before  (v3)                              after
dataset/                                  out/
  videos/observation.image/                 media/
    chunk-000/file-000.mp4  <- 206 eps         pack-0000.zip
  data/…  meta/…                                 …/episode_000000.mp4
                                                 …/episode_000001.mp4  <- one per episode
                                            data/…  meta/…  (copied: self-contained)
                                            video_index.json
                                              ep0  -> ref, [0 ns, 16.1 s)
                                              ep1  -> ref, [0 ns, 11.8 s)

before  (v2.1, frames in parquet)         after
data/episode_000000.parquet               media/pack-0000.zip
  action            float32[]               frames/observation.images.camera_front/…
  observation.…     struct<bytes, path>   data/episode_000000.parquet
                    ^ 31.6 GB of frames     action          float32[]
                                            observation.…   string  <- "zip://frames/…"

$ blobpack convert-lerobot ./dataset ./out
LeRobot v2.1 - 120 episodes, 120,000 frames @ 50 fps

  observation.images.camera_front    image    120,000 frames inside the table
      -> extracted to media/, column rewritten to blob references
      -> random reads stop pulling a whole row group
          bytes lossless, SCHEMA CHANGES
  data/*.parquet                     120 file(s), 31.6 GB
      -> rewritten
  meta/                              -> copied unchanged

Proceed? [y/N]

before                                    after
data.lance                                out/
  id       int64                            table.lance
  label    string                             id      int64
  img      large_binary   <- payloads         label   string
                                              img     string  <- "zip://img/…"
                                            media/pack-0000.zip
                                              img/000000000000.bin

$ blobpack convert-lance ./data.lance ./out
Lance dataset at data.lance - 500 rows

  image      binary column    500 values, JPEG, 146.5 KB mean
  sha256     binary column    500 values, unrecognized, 32 B mean
  embedding  binary column    500 values, unrecognized, 3.0 KB mean

Which columns hold payloads to extract?
  1) image       500 values, JPEG, 146.5 KB mean (146.5 KB-146.5 KB)
  2) sha256      500 values, unrecognized, 32 B mean (32 B-32 B)
  3) embedding   500 values, unrecognized, 3.0 KB mean (3.0 KB-3.0 KB)
Numbers, comma separated, or 'all' (blank aborts): 1

Specification

SPEC.md defines the format contract in one page: STORED zip shards as the default container, size and count limits, reference syntax, and immutability. This repository is one reference implementation of that contract. For timestamped media references and decoding, see MediaRef.

License

Apache-2.0