Skip to main content
image

Fast dataloader and conversion utility for webdataset tar shards. Rust core with Python bindings.

Built for streaming large video and image datasets, but handles any byte data.

Install

pip install webshart

What is this?

Webshart is a fast reader for webdataset tar files with separate JSON index files. This format enables random access to any file in the dataset without downloading the entire archive.

The indexed format provides massive performance benefits:

  • Random access: Jump to any file instantly
  • Selective downloads: Only fetch the files you need
  • True parallelism: Read from multiple shards simultaneously
  • Cloud-optimized: Works efficiently with HTTP range requests
  • Aspect bucketing: Optionally include image geometry hints width, height and aspect for the ability to bucket images by shape
  • Logical sample APIs: Treat image.ext + image.json pairs as one sample while still allowing raw file access
  • Caption metadata: Store captions under captions as strings, native JSON objects, or mixed lists of both
  • Custom DataLoader: Includes state dict methods on the DataLoader so that you can resume training deterministically
  • Rate-limit friendly: Local caching allows high-frequency random seeking without encountering storage provider rate limits
  • Instant start-up with pre-sorted aspect buckets

Growing ecosystem: While not all datasets use this format yet, you can easily create indices for any tar-based dataset (see below).

Quick Start

import webshart

# Find your dataset
dataset = webshart.discover_dataset(
    source="laion/conceptual-captions-12m-webdataset",
    # we're able to upload metadata separately so that we reduce load on huggingface infra.
    metadata="webshart/conceptual-captions-12m-webdataset-metadata",
)
print(f"Found {dataset.num_shards} shards")

loader = webshart.TarDataLoader(dataset)

# File-oriented access is still available.
files = dataset.list_files_in_shard(0)

# Sample-oriented access skips paired JSON sidecars.
samples = dataset.list_samples_in_shard(0)
entry = loader.load_sample(0, 0)
print(entry.path, entry.captions, entry.json_metadata)

Paired datasets

Two datasets can remain independently loadable while also exposing an opt-in join by logical sample key. This works especially well for preference, reference, and slider-training data stored in two subfolders of one repository:

paired = webshart.discover_paired_dataset(
    "webshart/suno-various-94k",
    left_subfolder="original",
    right_subfolder="covers",
)

print(paired.num_pairs)
print(paired.get_pair(0))

loader = webshart.PairedTarDataLoader(paired)
sample = loader.load_pair(0)
print(sample.key, sample.left, sample.right)

The normal contract is unchanged: calling discover_dataset(..., subfolder="original") or subfolder="covers" returns a standalone dataset. Pair indexing is lazy, preserves left-dataset order, and validates identical key sets by default. Pass strict=False to use only the intersection and inspect unmatched_left / unmatched_right.

max_file_size is a visibility limit for loader APIs. Files larger than the configured limit are omitted from iteration, batches, direct sample loading, and aspect buckets instead of being returned with empty data. Direct load_sample() calls return None for an oversized sample. The loader's list_samples_in_shard() returns dictionaries containing sample_idx and filename, so filtered listings retain the stable index required by load_sample().

Byte-bounded read-ahead

Use iter_prefetched() on a TarDataLoader or ParquetDataLoader to overlap sequential reads with work on the previous sample. Each iterator starts one background thread and yields entries in source order.

with loader.iter_prefetched(max_bytes=64 * 1024**2, max_items=128) as entries:
    for entry in entries:
        consume(entry)

The queue counts file bytes for tar and all requested binary payloads for Parquet, including secondary image columns. max_items also bounds the number of queued entries with empty or missing payloads. One oversized entry can pass through an empty queue on its own. The worker can have a read in flight beyond the queue, and the readers still use their own transport and decoding buffers.

Tar prefetch starts from a snapshot of the loader's current cursor and leaves its checkpoint state untouched. Parquet prefetch starts at the first shard by default. Pass shard_index, start, and stop to stream a physical row range within one shard. The context manager closes the iterator on early exit and waits for any in-flight read. Create the iterator in the process that consumes it.

Native transform cache

TransformCache stores caller-computed transform values in local SQLite files partitioned by shard. Supply a recipe fingerprint that covers the transform settings and value format, and identify each shard by its pinned source object and revision. Values are opaque bytes keyed by physical row number.

from webshart import TransformCache

cache = TransformCache("cache/transforms", "recipe-fingerprint")
shard = "capture/shard-00003.parquet@source-version"
try:
    cache.enqueue(shard, [(7, b"packed-transform"), (11, b"another-transform")])
    cache.flush()
    assert cache.read_batch([(shard, 11), (shard, 8), (shard, 11)]) == [
        b"another-transform", None, b"another-transform",
    ]
finally:
    cache.close()

enqueue hands a shard batch to the Rust writer. It applies backpressure when the queue is full and rejects batches above max_batch_bytes. A batch commits in one SQLite transaction. Reads preserve order and duplicates and report missing rows as None. Repeated writes keep the first committed value, so a recipe must compute deterministic values. flush and close wait for accepted writes and report writer errors. Open a separate handle in each process.

Set max_disk_bytes to a positive byte limit to enable automatic eviction. The limit applies to the shard databases and SQLite journals for all recipes and revisions in the cache directory. When the option is omitted, existing cache entries are retained indefinitely.

cache = TransformCache("cache/transforms", "recipe-fingerprint", max_disk_bytes=10 * 1024**3)

The cache checks its disk usage on open, periodically after writes, and when flush() or close() waits for writes. It removes oversized databases first, then evicts the least recently used idle databases until the cache fits. Reads of evicted entries return None, so callers can recompute those values. Active readers and transactions can temporarily keep the cache over budget. All processes sharing an eviction-enabled directory must use the same limit and an eviction-aware version of Webshart on storage that supports file locks.

Native Hugging Face Xet Downloads

Hugging Face files stored in Xet are read directly by webshart's Rust transport. Neither hf_xet nor huggingface_hub is needed for dataset reads. Non-Xet files and other HTTP servers continue to use ordinary HTTP downloads and byte ranges.

The transport supports full-shard caching, individual sample/sidecar reads, and batched range reads. It reconstructs files using concurrent 16 MiB windows, decodes uncompressed, LZ4, and byte-grouped LZ4 chunks, and handles CAS v2 multi-range responses and the documented v1 compatibility path. Buffers and request concurrency are bounded rather than proportional to the entire shard. Full-shard downloads validate the size and Hub-provided SHA-256 before entering the cache; partial reads validate ranges, chunk sizes, and reconstruction lengths. Transient requests are retried, expired credentials/transfer URLs are refreshed, and native Xet errors are reported rather than silently bypassing verification.

dataset = webshart.discover_dataset("webshart/pseudo-camera-10k-structured", subfolder="data")
dataset.enable_shard_cache("cache/shards", cache_limit_gb=25, parallel_downloads=4)
loader = webshart.TarDataLoader(dataset)
loader.prepare_shards_ahead(4)
sample = loader.load_sample(0, 0)

parallel_downloads limits concurrent whole shards in each cache; prefetch must be scheduled explicitly. Xet independently permits up to eight simultaneous chunk transfers per process, including multiple transfers within one shard. Set WEBSHART_XET_CONCURRENCY=1..64 before the first read to change that limit. load_sample() releases the Python GIL during I/O. Without a shard cache, only the requested sample ranges are reconstructed, not the entire archive.

Set WEBSHART_DISABLE_XET=1 to explicitly use the ordinary HTTP path instead. HF_ENDPOINT selects the Hub origin recognized by the Xet transport for custom resolve URLs; it does not change repository discovery's Hub endpoint.

Implemented from the public download protocol, authentication specification, and xorb format. Run the opt-in live test with cargo test --lib xet::tests::live_public_hub_shard -- --ignored --nocapture; it downloads a 287 MiB public shard, verifies its SHA-256, and compares partial reads with independent HTTP ranges. WEBSHART_TEST_LIVE_XET=1 .venv/bin/python -m pytest -q tests/test_xet_live.py also exercises the Python SDK, cache, batched reads, and ordinary HTTP opt-out with both Hugging Face Python libraries blocked from import.

Common Patterns

For real-world, working examples:

Creating Indices for / Converting Existing Datasets

Any tar-based webdataset can benefit from indexing! Webshart includes tools to generate indices:

A command-line tool that auto-discovers tars to process:

% webshart extract-metadata \
    --source laion/conceptual-captions-12m-webdataset \
    --destination laion_output/ \
    --checkpoint-dir ./laion_output/checkpoints \
    --max-workers 2 \
    --include-image-geometry

Or, if you prefer/require direct-integration to an existing Python application, use the API

Uploading Indices to HuggingFace

Once you've generated indices, share them with the community:

# Upload all JSON files to your dataset
huggingface-cli upload --repo-type=dataset \
    username/dataset-name \
    ./indices/ \
    --include "*.json" \
    --path-in-repo "indices/"

Or if you want to contribute to an existing dataset you don't own:

  1. Create a community dataset with indices: username/original-dataset-indices
  2. Upload the JSON files there
  3. Open a discussion on the original dataset suggesting they add the indices

Creating New Indexed Datasets

If you're creating a new dataset, generate indices during creation:

{
  "files": {
    "image_0001.webp": {"offset": 512, "length": 102400},
    "image_0002.webp": {"offset": 102912, "length": 98304},
    ...
  }
}

The JSON index should have the same name as the tar file (e.g., shard_0000.tar → shard_0000.json).

Caption layouts and sidecars

Webshart recognizes both JSON metadata sidecars and plain-text caption sidecars:

sample_0001.webp
sample_0001.json
sample_0002.webp
sample_0002.txt

Paired .json and .txt members are excluded from logical sample indexes. You can inspect the layout from shard metadata without downloading tar members:

layout = dataset.probe_caption_layout(max_shards=16)
print(layout["layout"])  # embedded, json_sidecar, txt_sidecar, mixed, or none

When metadata is extracted or loaded, sidecars are attached to their paired sample entries:

{
  "files": {
    "sample_0001.webp": {
      "offset": 512,
      "length": 102400,
      "width": 1024,
      "height": 1024,
      "aspect": 1.0,
      "json_path": "sample_0001.json",
      "json_offset": 103424,
      "json_length": 128,
      "captions": "a product photo on a white background",
      "json_metadata": {
        "caption": "a product photo on a white background"
      }
    },
    "sample_0001.json": {
      "offset": 103424,
      "length": 128
    }
  }
}

Use file-oriented APIs when you want every archive member, including sidecars:

dataset.list_files_in_shard(0)

reader = dataset.open_shard(0)
raw_file_bytes = reader.read_file(0)

Use sample-oriented APIs when you want training samples:

dataset.list_samples_in_shard(0)
dataset.get_shard_sample_count(0)

reader = dataset.open_shard(0)
image_bytes = reader.read_sample(0)
json_bytes = reader.read_sample_json(0)

entry = loader.load_sample(0, 0)
print(entry.path)
print(entry.captions)
print(entry.json_data)

# Direct caption lookup also handles paired .txt sidecars.
caption = loader.load_caption(0, 0)

Captions are canonicalized to the plural captions metadata key. The value may be a string, a native JSON object, a list containing either, or absent. Nested objects, arrays, numbers, and booleans inside caption objects retain their JSON types. entry.captions returns the complete value; entry.caption and loader.load_caption() return the first string or object.

The optimizer and caption coalescer recognize JSON objects and mixed caption lists in .txt sidecars. Plain text remains a string. JSON sidecars support native values under their caption fields and standalone caption objects.

webshart.write_captions_to_metadata(
    "shard_0000.json",
    {"sample_0001.webp": {"description": "a café", "elements": [{"bbox": [1, 2, 30, 40]}]}},
)
webshart.write_captions_to_metadata(
    "shard_0000.json",
    {
        "sample_0001.webp": "a short caption",
        "sample_0002": ["caption one", "caption two"],
    },
)

The writer updates existing webshart metadata JSON in place, removes old singular caption keys from updated samples, and leaves paired .json sidecar entries untouched.

Renaming metadata fields

Permanently rename a field across a dataset's JSON indexes:

dataset = webshart.discover_dataset("/path/to/dataset")
updated = dataset.field_rename("captions", "v1_captions")

loader = webshart.TarDataLoader(dataset)
entry = loader.load_sample(0, 0)
print(entry.metadata["v1_captions"])

field_rename(old_name, new_name, *, overwrite=False, destination=None) returns the number of file entries renamed. It moves top-level keys within each index's files entries, preserving values, custom fields, and dict/list index layouts. It processes one shard's metadata at a time without reading or rewriting tar payloads. Nested json_metadata and sidecar contents are left intact.

Missing fields are skipped; renaming a field to itself returns zero. Existing destination fields raise ValueError unless overwrite=True. Structural index fields such as offsets, lengths, and paths cannot be renamed. All indexes are validated before replacements begin, and each file replacement is atomic; an I/O failure during replacement can still leave earlier shards updated.

Local indexes are rewritten in place and their loaded/disk metadata caches are invalidated. Recreate existing loaders after renaming. Custom names are available through entry.metadata and loader.get_metadata(); entry.captions continues to represent the canonical caption field and may still read unchanged sidecars.

For remote indexes, pass destination="./renamed-metadata" to write a local export, then upload those JSON files separately. This also works for local datasets when an export is preferred. The dataset uses the exported indexes afterward, and tar files stay at their original locations.

Coalescing caption metadata

To avoid repeated .txt range reads, fold all sidecar captions into standard webshart metadata files. If metadata caching is enabled, omitting the destination persists the enriched indexes in webshart's cache:

dataset.enable_metadata_cache("cache/metadata", init_shard_count=0)
loader = webshart.TarDataLoader(dataset, load_file_data=False)
loader.coalesce_caption_metadata()

# Or create a portable export tree for copying or upload.
loader.coalesce_caption_metadata("caption-metadata")
webshart.upload_caption_metadata(
    "caption-metadata",
    "organization/dataset-metadata",
    hf_token="hf_...",
)

The CLI provides the same operation. --shard-cache-dir lets coalescing reuse full cached shards instead of issuing one range read per sidecar:

webshart optimize-captions \
  --source organization/dataset \
  --metadata organization/dataset-metadata \
  --destination caption-metadata \
  --shard-cache-dir cache/shards \
  --push-to-hub organization/dataset-metadata

optimize-captions expects existing .tar shards and webshart indexes. To fully repackage a repository of loose media plus .txt/.json sidecars, or a legacy SimpleTuner layout containing unindexed .tar archives whose member filenames are captions, use the rolling optimize-dataset command instead:

webshart optimize-dataset \
  --source stablellama/Qwen-Image-2512_samples \
  --push-to-hub stablellama/Qwen-Image-2512_samples \
  --output-prefix webshart \
  --max-shard-size-gb 1

The target is always a Hugging Face dataset repository. It may be the same repository as the source because generated files live under --output-prefix. The input layout is detected automatically. Loose sidecars are coalesced into metadata. Legacy tar members are repacked into bounded shards and their filename stems become captions, matching SimpleTuner's filename strategy (underscores become spaces). Remote legacy inputs use aligned HTTP ranges from the saved member offset and retain only the current output shard locally.

After each shard is indexed, its sidecar captions are embedded in the JSON index and the tar, index, and .webshart-optimize-state.json are uploaded in a single commit. The state records relative positions and conversion settings, never local absolute paths. For legacy tars this includes the source archive index and tar-block member offset, so a rerun resumes within an archive after the last committed output shard. Use --max-shards N to bound each worker invocation.

For a local-only conversion, replace --push-to-hub with a local destination:

webshart optimize-dataset \
  --source /datasets/loose-pairs \
  --destination /datasets/indexed \
  --max-shards 10

Plain-text sidecars are omitted from the tar after their captions are embedded. JSON sidecars are likewise coalesced: recognized caption fields become the canonical captions value and the complete object is retained as json_metadata in the index.

Hub reads accept hf_token= and also honor HF_TOKEN. This includes gated datasets and separately hosted metadata. Local discovery recursively pairs tar and JSON indexes, preserving their relative subdirectories.

Aspect Bucketing Samples

list_shard_aspect_buckets() is file-oriented and buckets any indexed file that has width and height.

For training pipelines, prefer list_shard_sample_aspect_buckets():

loader = webshart.TarDataLoader(dataset)
buckets = loader.list_shard_sample_aspect_buckets(
    [0],
    key="geometry-tuple",
    target_pixel_area=1024**2,
)[0]["buckets"]

for bucket_key, entries in buckets.items():
    for item in entries:
        virtual_id = f"webshart://0/{item['sample_idx']}/{item['filename']}"
        image = loader.load_sample(0, item["sample_idx"])

This uses logical samples from metadata.sample_range() / get_sample_by_index() and excludes paired JSON sidecars before bucketing. Each bucket entry includes sample_idx, so callers can build stable IDs and load images directly with loader.load_sample(shard_idx, sample_idx).

Parquet Payloads With Custom Metadata

The Parquet API is available starting with webshart 0.7.0. Existing tar APIs keep their behavior. Use ParquetDataLoader, not TarDataLoader or BucketDataLoader, for Parquet sources:

dataset = webshart.discover_dataset(
    "InternVL-U/ScaleEdit-12M",
    format="parquet",
    metadata="webshart/ScaleEdit-12M-metadata",
)
loader = webshart.ParquetDataLoader(
    dataset, image_columns=["source_image", "edited_image"]
)
metadata = loader.get_metadata(0, 0)
sample = loader.load_sample(0, 0)
edited_bytes = sample.data
source_bytes = sample.payloads["source_image"]
instruction = sample.captions

Local directories work too. Without separate metadata, specify the source column names with image_column, caption_column, width_column, and height_column. discover_parquet_dataset(...) is equivalent to discovery with format="parquet". Hub discovery requires webshart[hub]; payload decoding is native Rust and does not require PyArrow, datasets, or hf_xet. Hub payload ranges reuse webshart's native Xet transport, with HTTP Range for non-Xet files.

Counting Parquet image rows

len(dataset) counts physical rows. Call dataset.count_rows(exclude_null_images=True) when you need to count rows with embedded image bytes. Pass shard_index to limit either count to one shard. A zero-length byte string is present, and the count leaves physical row indices unchanged.

The image count uses footer statistics where available. For row groups without null statistics, the reader scans the selected image column in bounded batches. The count applies to embedded bytes, including the bytes field in an image struct.

Projected metadata reads

Read selected physical rows and columns without materializing every row's metadata in the shard:

rows = dataset.read_metadata_rows(0, [12, 3, 12], fields=["width", "height"])
samples = loader.load_samples([(0, 12), (0, 3), (0, 12)], metadata_fields=[])
for entry in loader.iter_samples_in_shard(0, metadata_fields=["width", "height"], batch_size=256):
    print(entry["sample_idx"], entry["width"], entry["height"])

Requested order and duplicates are preserved. Every metadata result includes source_path and source_row; fields=[] returns only that identity. captions, width, and height resolve configured column aliases and custom overrides. Payload columns are excluded. metadata_fields=[] on load_samples loads image payloads without captions or other optional metadata. Omitting fields or metadata_fields returns all non-payload metadata.

Projected sample listings still read capture status fields and omit failed captures. Aspect buckets accept metadata_fields too and add the dimensions needed for bucketing. list_samples_in_shard materializes the resulting list; iter_samples_in_shard bounds each projected read by batch_size.

Capture manifests and custom metadata overlays use temporary SQLite indexes. Discovery streams and verifies the capture JSONL manifest without retaining a Python dictionary per shard or sending a HEAD request per object. For capture manifests, dataset.physical_row_count and len(dataset) return the published physical row count without opening payloads; this includes failed fetches. physical_row_count is None for other discovery modes.

Custom overlays are fully validated when their index is built, including sparse row identity, duplicate rows, replacement coverage, and optional checksums. Only requested source columns are decoded by the native reader. Index and reader counts follow metadata_cache_size; large selected dictionaries can still cost substantial memory, and JSON companions retain their 16 MiB limit. Cached payload readers also follow the shard cache's byte budget, while allowing one oversized shard to be opened.

For a persisted dataset view, save dataset.source_revision_for_shard(index) alongside each shard's accepted rows, then call dataset.bind_source_revisions(lookup) on the reopened dataset. lookup(index) returns the saved revision or None for an unpinned shard. New reader opens reject changed content: local files use SHA-256, Hub files use the pinned commit, and S3 objects use ETags. Unhashed custom overlays are included in the revision. Binding a lookup clears existing reader and metadata caches.

dataset.io_stats() reports logical S3 HEAD, range, download, and listing operations, plus transferred range/download bytes; SDK retries are internal. It returns an empty dictionary for other transports. These counters include manifest I/O. Per-shard sha256 fields in a capture manifest are compared with S3's sha256 object metadata when that metadata is present; range reads retain their ETag checks.

Captured images on S3

Native S3 access and batched row-group reads require webshart 0.8.0 or later.

S3-compatible Parquet sources use native Rust AWS signing, connection pooling, ETag-pinned range reads, and streamed shard downloads. No Python S3 transfer library is required. Supply an endpoint without the bucket name:

dataset = webshart.discover_parquet_dataset(
    "s3://training-images",
    manifest="combined/dataset.json",
    s3_options={"endpoint_url": "https://storage.example", "max_connections": 32},
)
dataset.enable_shard_cache("cache/parquet", cache_limit_gb=25, parallel_downloads=4)
loader = webshart.ParquetDataLoader(dataset)
samples = loader.load_samples([(0, 12), (0, 3), (0, 12)], parallel_reads=4)

Omit manifest to list Parquet objects below the source prefix. A capture descriptor has format: "parquet-image-capture", version: 1 or 2, and a files_manifest pointing to source-relative JSONL entries containing key, size_bytes, and num_rows. An optional manifest_sha256 verifies the JSONL contents. Discovery uses manifest sizes instead of sending one HEAD per part. Object size and row counts are checked when a part is opened.

Credentials can come from AWS environment variables, web identity, or container and instance roles. Explicit aws_access_key_id, aws_secret_access_key, aws_session_token, and region_name are also accepted in s3_options. allow_http=True is available for local S3 test servers. Shared AWS profile files are not read by this transport.

Without a shard cache, only selected Parquet ranges are fetched. The shard cache streams whole objects to disk and reuses webshart's download locks and eviction. Cache entries are scoped by endpoint, bucket, object key, and ETag. A cached shard is locked while it is being read, so another process can evict it while a reader is idle. A dataset reuses the size and ETag it verified on first open. Reopening an evicted reader uses conditional S3 reads and downloads with that ETag. Reopen the dataset to verify the latest object revision.

Metadata preparation coalesces selected column chunks into a bounded 64 MiB cache and leaves image columns unread. Whole-shard downloads begin on payload access. Network I/O and Parquet decoding release the GIL. Batch requests share each selected row group, use bounded native concurrency, and preserve input order and duplicates.

loader.get_metadata(shard) returns logical sample metadata; loader.list_samples_in_shard(shard) and loader.list_shard_sample_aspect_buckets([shard]) omit failed capture rows. Physical row indices stay unchanged. dataset.fingerprint identifies the source configuration and catalog snapshot for downstream metadata caches. Captured bytes are read directly, without recompression or fetching the original URL. Applications decode the returned bytes; JPEG XL requires a decoder such as pillow-jxl-plugin.

Online images from URL columns

URL-only Parquet datasets such as Recap-DataComp-1B can use the same metadata overlays without storing a second copy of the images:

dataset = webshart.discover_parquet_dataset(
    "UCSC-VLAA/Recap-DataComp-1B",
    subfolder="data/train_data",
    image_url_column="url",
    caption_column="re_caption",
)
loader = webshart.ParquetDataLoader(
    dataset,
    fetch_external_urls=True,
    url_options=webshart.URLImageOptions(timeout=10, retries=1, max_bytes=16 * 1024**2),
    on_url_error="return",
)
sample = loader.load_sample(0, 0)
if sample.error is None and sample.data is not None:
    image_bytes, caption = sample.data, sample.captions

Discovery and get_metadata() never fetch external images. Without fetch_external_urls=True, URL samples have data=None. With it enabled, Rust fetches HTTP(S) bytes on demand using a reusable connection pool, bounded response sizes, a timeout covering each request and its body, and retries for transient failures. Hub credentials are never passed to image hosts. Image bytes are neither persisted nor uploaded; repeated loads fetch the URL again. The loader returns raw bytes and does not decode or validate image contents.

Failures raise URLImageError by default. on_url_error="return" yields data=None and an error string instead, preserving the physical row identity and iteration order. A missing URL is an invalid_url failure. payloads retains the original URL, and original captions and hash columns remain in metadata. An existing sha256 is not assumed to hash the currently served image bytes.

URL-source metadata reads use a bounded native Parquet cache to avoid repeated small network reads. url_metadata_cache_bytes defaults to 128 MiB per cached source shard; larger files use the existing page reader. Set it to zero to disable this cache. This caches source Parquet bytes, never fetched image bytes. The geometry scanner projects only the URL column and does not fill this cache.

Scanning image dimensions and availability

Install webshart[scan] to build metadata overlays. The Rust scanner fetches images concurrently, reads dimensions from their headers, computes downloaded byte hashes, and by default discards image bytes. Python writes checkpointed Parquet metadata. Header inspection is not a full pixel decode or image-quality check.

Start with a bounded pilot and pin the source revision for repeatable resumes:

webshart scan-url-metadata \
  --source UCSC-VLAA/Recap-DataComp-1B \
  --revision 457c44d98651bcfdfb3cc8695f5e60a0d2705e78 \
  --subfolder data/train_data \
  --url-column url --caption-column re_caption \
  --destination ./recap-metadata \
  --workers 128 --batch-size 1000 \
  --timeout 10 --retries 1 --max-bytes 16777216 \
  --max-samples 1000

--workers bounds simultaneous native requests, not Python threads. Peak image buffer memory scales with concurrency times --max-bytes; metadata batches are bounded by --batch-size. Checkpoints are separate per source shard. Re-running with the same source and fetch options resumes after committed batches; --max-samples limits new attempts in each invocation. Worker count and batch size may change. An interrupted batch is retried. Previously recorded failures are retained; use a fresh destination to reassess them. Do not run two writers against the same destination. For distributed scans, use disjoint --start-shard/--stop-shard ranges and separate output directories.

Add --capture-images to also write image-bearing Parquet shards under data/. Each row contains image: {bytes, path}, captions, url, physical source_shard and source_row, and the scan metadata. Failed probes retain null images and their failure status. Filter url_status == "ok" for training. Rust stages successful downloads while Python writes bounded Arrow row groups; images are fetched only once per successful batch.

Optional --recompress-images (SDK: recompress_images=True) losslessly recompresses captured JPEGs to JPEG XL. Install the reference cjxl and djxl tools on PATH; missing tools fail the scan before downloading a batch. Rust invokes the codecs without a shell, limits each codec to one thread, reconstructs the JPEG, and compares the reconstructed file byte for byte with the download. Only smaller, verified results replace originals. Other formats, codec failures, and conversions that do not save space retain the original bytes. This is JPEG bitstream recompression, not lossy pixel re-encoding.

--recompress-workers (SDK: recompress_workers, default 2) separately bounds encoding concurrency per scan invocation. Codec calls have a 60-second timeout each. Rust performs staging, hashing, verification, and cleanup with the Python GIL released. The codec implementation is supplied by the external libjxl tools. Captured rows include stored_format (jxl or original), stored_bytes, stored_sha256, and recompression_status; the existing download_* fields always describe the original response. Failed downloads have null storage fields. JXL bytes require a compatible training decoder or JPEG reconstruction with djxl; webshart returns the stored bytes unchanged. Enabling recompression requires a fresh scan destination so a resume cannot silently mix policies.

To upload each completed shard and discard its local image copy, add:

--capture-images --upload-repo owner/dataset --no-local-images

The destination repository must already exist. Uploads use huggingface_hub authentication and endpoint configuration; --upload-endpoint overrides only the upload endpoint. --upload-repo-type model supports repositories exposed through the model API. The SDK equivalents are capture_images=True, upload_repo="owner/dataset", keep_local_images=False, and optional upload_endpoint, upload_token, and upload_repo_type arguments to scan_url_metadata. Explicit upload credentials do not change source reads.

This mode needs temporary disk space for one download batch and its Parquet shard per process (bounded by batch size times the maximum image size, plus encoding overhead). It retains local metadata and checkpoints. Images are removed only after a successful Hub commit and a durable local checkpoint. A failed upload retains the completed batch for retry without refetching. Resuming trusts recorded Hub commit receipts; keep the uploaded files intact. The destination data/ files are self-contained image/caption Parquet shards; the metadata overlay still refers to the original URL source. Local captured shards can be loaded with discover_parquet_dataset("./recap-metadata/data"). For a remote captured repository, use subfolder="data" and optionally endpoint and repo_type="model" in discover_parquet_dataset. Use a fresh scan destination when enabling capture on an existing metadata-only scan: discarded image bytes must be downloaded again. Use separate upload repositories for different sources or revisions, and disjoint shard ranges for concurrent writers to one repository.

For a separate uploader, give each producer a distinct child scan directory and matching staging directory, then add --defer-uploads --no-local-images with --upload-repo. For example, metadata in ./scans/000000 can stage bytes in ./buffer/000000 using --staging-directory. Downloads, native recompression, and image Parquet writing continue without waiting for Hub uploads. --min-staging-free-bytes pauses producers when the staging filesystem is low on free space. Keep staging directories exclusive to their corresponding scans.

A separate process can periodically run:

webshart upload-capture-queue \
  --destination ./scans --staging-directory ./buffer \
  --upload-repo owner/dataset --min-bytes 1099511627776

The uploader consumes completed child scans once at least 1 TiB is pending. It retains individual Parquet files and uploads groups of at most 64 files per commit, with four transfer threads by default. Only one uploader may operate on a queue. The SDK equivalent is upload_capture_queue; its byte threshold, commit size, and transfer concurrency are configurable. Run with --flush after all producers finish to upload the final remainder. If resuming an interrupted batch, repeat until the returned pending_bytes is zero. Capture completion alone is not proof of upload completion.

Active batch state and commit receipts remain with the scan metadata. A small receipt marker is committed alongside each remote file group, allowing retries to recognize a successful commit even if its acknowledgment was lost. Staged files are removed after the receipt and checkpoints are saved. Preserve both metadata and staging across restarts; missing pending image files cause an error rather than silently skipping rows. Deferred mode is part of scan identity; existing immediate-upload scans require an explicit checkpoint migration or a fresh destination before changing modes.

For independent workers claiming source shards dynamically, use the object-store shard queue. Conditional writes give each attempt a separate output prefix and prevent a replaced owner from publishing its result. Keep renewing claims until their outputs and manifest are uploaded; the shard queue coordinates ownership separately from the upload queue.

The resulting webshart-parquet.json and metadata/ files can be used locally or hosted together in a metadata-only Hub repository. Keep scan-state.json, checkpoints/, and parts/ locally for resuming. Each completed invocation publishes an overlay covering the rows attempted so far, including partial shards; unscanned rows retain their original metadata.

dataset = webshart.discover_parquet_dataset(
    "UCSC-VLAA/Recap-DataComp-1B",
    metadata="./recap-metadata",
)
loader = webshart.ParquetDataLoader(dataset, fetch_external_urls=True)
metadata = loader.get_metadata(0, 0)
if metadata.get("url_status") == "ok":
    image = loader.load_sample(0, 0)

Overlay rows include width, height, url_status, url_http_status, download_bytes, and download_sha256. scan-summary.json separates successful geometry reads, HTTP status codes, timeouts, network errors, size limits and invalid images. These are observations from one scan: access denial, temporary failures, and unsupported formats are not proof of permanent link rot. A prefix pilot is not a representative estimate of availability across the whole dataset. Image URLs can change even when the Parquet source revision is pinned, and a successful scan does not guarantee a later training request will succeed.

Metadata manifest format

Pass a local metadata directory or Hub dataset repository containing webshart-parquet.json. The source files stay unchanged. For example:

{
  "format": "webshart-parquet",
  "version": 1,
  "source": {"repo_id": "owner/source", "revision": "<40-character commit SHA>"},
  "columns": {"image": "image", "captions": "instruction", "width": "width", "height": "height"},
  "metadata_mode": "overlay",
  "shards": [{
    "path": "data/part-000.parquet",
    "num_rows": 2,
    "metadata_path": "custom/part-000.json"
  }]
}

For URL sources, set "image_url": "url" in columns instead of "image". This declares the payload column but does not enable network loading; the loader still requires fetch_external_urls=True. Overlays cannot replace payload URLs.

The JSON metadata file is a list of row objects:

[
  {"source_row": 1, "captions": [{"scene": "sunset"}, "warm lighting"], "rating": 5},
  {"source_row": 0, "captions": {"scene": "forest", "lighting": "soft"}}
]

Alternatively, metadata_path can name a Parquet file with a source_row integer column and arbitrary custom columns. Row order and source IDs do not matter: the join uses the source shard path plus zero-based physical row. Optional source_path values must match the manifest. Duplicates, invalid row positions, unsafe paths, mismatched file sizes/row counts, and incorrect optional metadata_sha256 checksums fail explicitly. size, when present, is the source Parquet file size in bytes. Remote manifests must pin their source commit; metadata_revision can additionally pin the metadata repository commit.

overlay merges custom values over source non-binary columns and permits sparse overrides. Custom captions, width, and height override configured aliases. replace reads metadata only from the companion file and requires every row to be present. It avoids reading upstream metadata columns. Payload columns cannot be overridden. Metadata is cached per shard with a bounded LRU (default: one shard; adjust metadata_cache_size). Recreate the dataset after changing files. Custom companion rows are indexed on temporary disk and read back by physical row. Overlay indexes are retained up to metadata_cache_size; eviction closes the least recently used index.

Parquet structs, lists, and maps retain their native Python representations. For mixed strings/objects that Parquet cannot represent in one homogeneous column, store JSON text and declare "json_columns": ["captions"] in the manifest (or discovery options). Only explicitly declared columns are parsed; ordinary caption strings are not guessed or stringified. Decimal values remain decimal.Decimal; date/time values use their Parquet integer units.

Streaming and access costs

loader.iter_shard(shard_index, start=0, stop=None) streams selected image columns page by page, reusing dictionaries. load_sample(shard_index, row_index) skips preceding row groups but scans earlier rows within the selected group. Unlike indexed tar, compressed Parquet does not provide constant-cost per-image byte offsets. Prefer sequential iteration for large row groups. Memory scales with selected columns' decoded pages/dictionaries, not an entire shard; oversized dictionaries can still require hundreds of MiB. The max_page_bytes discovery option limits individual compressed page/footer reads (default 512 MiB), not total or decompressed memory.

Binary image columns and Hugging Face {bytes, path} image structs are supported. Missing bytes produce sample.data is None; original structs remain in sample.payloads. External URLs/path-only images are never fetched automatically. Keep URL/hash metadata and apply your own retrieval policy. The loader does not rewrite or download whole source shards, and does not expose tar-only byte-offset, bucket, shard-cache, or field-rewrite APIs.

Why is indexed tar fast?

Problem: Standard tar files require sequential reading. To get file #10,000, you must read through files #1-9,999 first.

Solution: The indexed format stores byte offsets and sample metadata in a separate JSON file, enabling:

  • HTTP range requests for any file
  • True random access over network
  • Parallel reads from multiple shards
  • Large scale, aspect-bucketed datasets
  • No wasted bandwidth

The Rust implementation provides:

  • Real parallelism (no Python GIL)
  • Zero-copy operations where possible
  • Efficient HTTP connection pooling
  • Optimized tokio async runtime
  • Optional local caching for metadata and shards
  • Fast aspect bucketing for image data

Datasets Using This Format

I discovered after creating this library that cheesechaser is the origin of the indexed tar format, which webshart has formalised and extended to include aspect bucketing support.

Requirements

  • Python 3.12+
  • Linux/macOS/Windows

Roadmap

  • image decoding is currently not handled by this library, but it will be added with zero-copy.
  • more informative API for caching and other Rust implementation details
  • multi-gpu/multi-node friendly dataloader

Projects using webshart

  • CaptionFlow uses this library to solve memory use and seek performance issues typical to webdatasets

License

MIT

Metadata

Release files for webshart 0.9.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for webshart 0.9.2
File Size Uploaded Provenance Actions
webshart-0.9.2.tar.gz 273.1 kB Without provenance Details for webshart-0.9.2.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for webshart 0.9.2
File Actions
webshart-0.9.2-cp312-abi3-win_amd64.whl CPython 3.12 abi3 Windows x86-64 Details for webshart-0.9.2-cp312-abi3-win_amd64.whl
webshart-0.9.2-cp312-abi3-manylinux_2_39_x86_64.whl CPython 3.12 abi3 Linux glibc 2.39+ x86-64 Details for webshart-0.9.2-cp312-abi3-manylinux_2_39_x86_64.whl
webshart-0.9.2-cp312-abi3-manylinux_2_39_aarch64.whl CPython 3.12 abi3 Linux glibc 2.39+ ARM64 Details for webshart-0.9.2-cp312-abi3-manylinux_2_39_aarch64.whl
webshart-0.9.2-cp312-abi3-manylinux_2_35_x86_64.whl CPython 3.12 abi3 Linux glibc 2.35+ x86-64 Details for webshart-0.9.2-cp312-abi3-manylinux_2_35_x86_64.whl
webshart-0.9.2-cp312-abi3-macosx_11_0_arm64.whl CPython 3.12 abi3 macOS 11.0+ ARM64 Details for webshart-0.9.2-cp312-abi3-macosx_11_0_arm64.whl

Total release size: 42.1 MB

Release files / webshart-0.9.2.tar.gz

Download URL webshart-0.9.2.tar.gz
Size 273.1 kB
Tags Source
SHA-256 checksum
How to use checksums
548e4b1116229cdb8d99ff9bf2d010cf34f7ae3f692f53dba5b4d1997ef000fa
BLAKE2b-256 checksum
How to use checksums
81c48f250339c84f31a9a35463152fb752e7ffc9d5443043e2a09fb614c5c1ea
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Release files / webshart-0.9.2-cp312-abi3-win_amd64.whl

Download URL webshart-0.9.2-cp312-abi3-win_amd64.whl
Size 8.3 MB
Tags CPython 3.12 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
30308d00b93867c58ee55a19529589bdc5363afdb1760e732aafcf15ea7d65fd
BLAKE2b-256 checksum
How to use checksums
8242c3f6185ae024afa7dfd23810706933c4970a1677aac814c01fcb206db424
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Release files / webshart-0.9.2-cp312-abi3-manylinux_2_39_x86_64.whl

Download URL webshart-0.9.2-cp312-abi3-manylinux_2_39_x86_64.whl
Size 8.6 MB
Tags CPython 3.12 Linux glibc 2.39+ x86-64 abi3
SHA-256 checksum
How to use checksums
8661554eb6bbccc0a4fd46734b3d81b777a746685c6e229bd8462b58f4533aee
BLAKE2b-256 checksum
How to use checksums
d97fc60c9cdb65eb71c53e344287e6a4211f7d24e04d2b5f9362ec5ed576189a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Release files / webshart-0.9.2-cp312-abi3-manylinux_2_39_aarch64.whl

Download URL webshart-0.9.2-cp312-abi3-manylinux_2_39_aarch64.whl
Size 8.2 MB
Tags CPython 3.12 Linux glibc 2.39+ ARM64 abi3
SHA-256 checksum
How to use checksums
47f6cfdd4d851e41c81251353373bbe46b36f179b01e197f01d2c41141be34b1
BLAKE2b-256 checksum
How to use checksums
9ce8fedbe776a5ae6188db5fe15d416c28c3f6a0d1614d458da6fe35c771512b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Release files / webshart-0.9.2-cp312-abi3-manylinux_2_35_x86_64.whl

Download URL webshart-0.9.2-cp312-abi3-manylinux_2_35_x86_64.whl
Size 8.6 MB
Tags CPython 3.12 Linux glibc 2.35+ x86-64 abi3
SHA-256 checksum
How to use checksums
acb6de1d2202f232c90493644632e41e6d842736200fdd17c92bc0bfd8e254cf
BLAKE2b-256 checksum
How to use checksums
82618116e37a6c69a96fd771d0940d601882759b44550d75d398ff27cfa2d0dd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Release files / webshart-0.9.2-cp312-abi3-macosx_11_0_arm64.whl

Download URL webshart-0.9.2-cp312-abi3-macosx_11_0_arm64.whl
Size 8.1 MB
Tags CPython 3.12 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
a26013496702de58fe197b163fd58e570ec797a798f0d90125470240a1726f10
BLAKE2b-256 checksum
How to use checksums
e53da302688eb7f4b39c4e6449e84053675b20d7d2a48b44fbe1b041d519c68d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Security for webshart 0.9.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page