Skip to main content
image

Fast dataloader and conversion utility for webdataset tar shards. Rust core with Python bindings.

Built for streaming large video and image datasets, but handles any byte data.

Install

pip install webshart

What is this?

Webshart is a fast reader for webdataset tar files with separate JSON index files. This format enables random access to any file in the dataset without downloading the entire archive.

The indexed format provides massive performance benefits:

  • Random access: Jump to any file instantly
  • Selective downloads: Only fetch the files you need
  • True parallelism: Read from multiple shards simultaneously
  • Cloud-optimized: Works efficiently with HTTP range requests
  • Aspect bucketing: Optionally include image geometry hints width, height and aspect for the ability to bucket images by shape
  • Logical sample APIs: Treat image.ext + image.json pairs as one sample while still allowing raw file access
  • Caption metadata: Store captions under captions as strings, native JSON objects, or mixed lists of both
  • Custom DataLoader: Includes state dict methods on the DataLoader so that you can resume training deterministically
  • Rate-limit friendly: Local caching allows high-frequency random seeking without encountering storage provider rate limits
  • Instant start-up with pre-sorted aspect buckets

Growing ecosystem: While not all datasets use this format yet, you can easily create indices for any tar-based dataset (see below).

Quick Start

import webshart

# Find your dataset
dataset = webshart.discover_dataset(
    source="laion/conceptual-captions-12m-webdataset",
    # we're able to upload metadata separately so that we reduce load on huggingface infra.
    metadata="webshart/conceptual-captions-12m-webdataset-metadata",
)
print(f"Found {dataset.num_shards} shards")

loader = webshart.TarDataLoader(dataset)

# File-oriented access is still available.
files = dataset.list_files_in_shard(0)

# Sample-oriented access skips paired JSON sidecars.
samples = dataset.list_samples_in_shard(0)
entry = loader.load_sample(0, 0)
print(entry.path, entry.captions, entry.json_metadata)

Paired datasets

Two datasets can remain independently loadable while also exposing an opt-in join by logical sample key. This works especially well for preference, reference, and slider-training data stored in two subfolders of one repository:

paired = webshart.discover_paired_dataset(
    "webshart/suno-various-94k",
    left_subfolder="original",
    right_subfolder="covers",
)

print(paired.num_pairs)
print(paired.get_pair(0))

loader = webshart.PairedTarDataLoader(paired)
sample = loader.load_pair(0)
print(sample.key, sample.left, sample.right)

The normal contract is unchanged: calling discover_dataset(..., subfolder="original") or subfolder="covers" returns a standalone dataset. Pair indexing is lazy, preserves left-dataset order, and validates identical key sets by default. Pass strict=False to use only the intersection and inspect unmatched_left / unmatched_right.

max_file_size is a visibility limit for loader APIs. Files larger than the configured limit are omitted from iteration, batches, direct sample loading, and aspect buckets instead of being returned with empty data. Direct load_sample() calls return None for an oversized sample. The loader's list_samples_in_shard() returns dictionaries containing sample_idx and filename, so filtered listings retain the stable index required by load_sample().

Native Hugging Face Xet Downloads

Hugging Face files stored in Xet are read directly by webshart's Rust transport. Neither hf_xet nor huggingface_hub is needed for dataset reads. Non-Xet files and other HTTP servers continue to use ordinary HTTP downloads and byte ranges.

The transport supports full-shard caching, individual sample/sidecar reads, and batched range reads. It reconstructs files using concurrent 16 MiB windows, decodes uncompressed, LZ4, and byte-grouped LZ4 chunks, and handles CAS v2 multi-range responses and the documented v1 compatibility path. Buffers and request concurrency are bounded rather than proportional to the entire shard. Full-shard downloads validate the size and Hub-provided SHA-256 before entering the cache; partial reads validate ranges, chunk sizes, and reconstruction lengths. Transient requests are retried, expired credentials/transfer URLs are refreshed, and native Xet errors are reported rather than silently bypassing verification.

dataset = webshart.discover_dataset("webshart/pseudo-camera-10k-structured", subfolder="data")
dataset.enable_shard_cache("cache/shards", cache_limit_gb=25, parallel_downloads=4)
loader = webshart.TarDataLoader(dataset)
loader.prepare_shards_ahead(4)
sample = loader.load_sample(0, 0)

parallel_downloads limits concurrent whole shards in each cache; prefetch must be scheduled explicitly. Xet independently permits up to eight simultaneous chunk transfers per process, including multiple transfers within one shard. Set WEBSHART_XET_CONCURRENCY=1..64 before the first read to change that limit. load_sample() releases the Python GIL during I/O. Without a shard cache, only the requested sample ranges are reconstructed, not the entire archive.

Set WEBSHART_DISABLE_XET=1 to explicitly use the ordinary HTTP path instead. HF_ENDPOINT selects the Hub origin recognized by the Xet transport for custom resolve URLs; it does not change repository discovery's Hub endpoint.

Implemented from the public download protocol, authentication specification, and xorb format. Run the opt-in live test with cargo test --lib xet::tests::live_public_hub_shard -- --ignored --nocapture; it downloads a 287 MiB public shard, verifies its SHA-256, and compares partial reads with independent HTTP ranges. WEBSHART_TEST_LIVE_XET=1 .venv/bin/python -m pytest -q tests/test_xet_live.py also exercises the Python SDK, cache, batched reads, and ordinary HTTP opt-out with both Hugging Face Python libraries blocked from import.

Common Patterns

For real-world, working examples:

Creating Indices for / Converting Existing Datasets

Any tar-based webdataset can benefit from indexing! Webshart includes tools to generate indices:

A command-line tool that auto-discovers tars to process:

% webshart extract-metadata \
    --source laion/conceptual-captions-12m-webdataset \
    --destination laion_output/ \
    --checkpoint-dir ./laion_output/checkpoints \
    --max-workers 2 \
    --include-image-geometry

Or, if you prefer/require direct-integration to an existing Python application, use the API

Uploading Indices to HuggingFace

Once you've generated indices, share them with the community:

# Upload all JSON files to your dataset
huggingface-cli upload --repo-type=dataset \
    username/dataset-name \
    ./indices/ \
    --include "*.json" \
    --path-in-repo "indices/"

Or if you want to contribute to an existing dataset you don't own:

  1. Create a community dataset with indices: username/original-dataset-indices
  2. Upload the JSON files there
  3. Open a discussion on the original dataset suggesting they add the indices

Creating New Indexed Datasets

If you're creating a new dataset, generate indices during creation:

{
  "files": {
    "image_0001.webp": {"offset": 512, "length": 102400},
    "image_0002.webp": {"offset": 102912, "length": 98304},
    ...
  }
}

The JSON index should have the same name as the tar file (e.g., shard_0000.tar → shard_0000.json).

Caption layouts and sidecars

Webshart recognizes both JSON metadata sidecars and plain-text caption sidecars:

sample_0001.webp
sample_0001.json
sample_0002.webp
sample_0002.txt

Paired .json and .txt members are excluded from logical sample indexes. You can inspect the layout from shard metadata without downloading tar members:

layout = dataset.probe_caption_layout(max_shards=16)
print(layout["layout"])  # embedded, json_sidecar, txt_sidecar, mixed, or none

When metadata is extracted or loaded, sidecars are attached to their paired sample entries:

{
  "files": {
    "sample_0001.webp": {
      "offset": 512,
      "length": 102400,
      "width": 1024,
      "height": 1024,
      "aspect": 1.0,
      "json_path": "sample_0001.json",
      "json_offset": 103424,
      "json_length": 128,
      "captions": "a product photo on a white background",
      "json_metadata": {
        "caption": "a product photo on a white background"
      }
    },
    "sample_0001.json": {
      "offset": 103424,
      "length": 128
    }
  }
}

Use file-oriented APIs when you want every archive member, including sidecars:

dataset.list_files_in_shard(0)

reader = dataset.open_shard(0)
raw_file_bytes = reader.read_file(0)

Use sample-oriented APIs when you want training samples:

dataset.list_samples_in_shard(0)
dataset.get_shard_sample_count(0)

reader = dataset.open_shard(0)
image_bytes = reader.read_sample(0)
json_bytes = reader.read_sample_json(0)

entry = loader.load_sample(0, 0)
print(entry.path)
print(entry.captions)
print(entry.json_data)

# Direct caption lookup also handles paired .txt sidecars.
caption = loader.load_caption(0, 0)

Captions are canonicalized to the plural captions metadata key. The value may be a string, a native JSON object, a list containing either, or absent. Nested objects, arrays, numbers, and booleans inside caption objects retain their JSON types. entry.captions returns the complete value; entry.caption and loader.load_caption() return the first string or object.

The optimizer and caption coalescer recognize JSON objects and mixed caption lists in .txt sidecars. Plain text remains a string. JSON sidecars support native values under their caption fields and standalone caption objects.

webshart.write_captions_to_metadata(
    "shard_0000.json",
    {"sample_0001.webp": {"description": "a café", "elements": [{"bbox": [1, 2, 30, 40]}]}},
)
webshart.write_captions_to_metadata(
    "shard_0000.json",
    {
        "sample_0001.webp": "a short caption",
        "sample_0002": ["caption one", "caption two"],
    },
)

The writer updates existing webshart metadata JSON in place, removes old singular caption keys from updated samples, and leaves paired .json sidecar entries untouched.

Renaming metadata fields

Permanently rename a field across a dataset's JSON indexes:

dataset = webshart.discover_dataset("/path/to/dataset")
updated = dataset.field_rename("captions", "v1_captions")

loader = webshart.TarDataLoader(dataset)
entry = loader.load_sample(0, 0)
print(entry.metadata["v1_captions"])

field_rename(old_name, new_name, *, overwrite=False, destination=None) returns the number of file entries renamed. It moves top-level keys within each index's files entries, preserving values, custom fields, and dict/list index layouts. It processes one shard's metadata at a time without reading or rewriting tar payloads. Nested json_metadata and sidecar contents are left intact.

Missing fields are skipped; renaming a field to itself returns zero. Existing destination fields raise ValueError unless overwrite=True. Structural index fields such as offsets, lengths, and paths cannot be renamed. All indexes are validated before replacements begin, and each file replacement is atomic; an I/O failure during replacement can still leave earlier shards updated.

Local indexes are rewritten in place and their loaded/disk metadata caches are invalidated. Recreate existing loaders after renaming. Custom names are available through entry.metadata and loader.get_metadata(); entry.captions continues to represent the canonical caption field and may still read unchanged sidecars.

For remote indexes, pass destination="./renamed-metadata" to write a local export, then upload those JSON files separately. This also works for local datasets when an export is preferred. The dataset uses the exported indexes afterward, and tar files stay at their original locations.

Coalescing caption metadata

To avoid repeated .txt range reads, fold all sidecar captions into standard webshart metadata files. If metadata caching is enabled, omitting the destination persists the enriched indexes in webshart's cache:

dataset.enable_metadata_cache("cache/metadata", init_shard_count=0)
loader = webshart.TarDataLoader(dataset, load_file_data=False)
loader.coalesce_caption_metadata()

# Or create a portable export tree for copying or upload.
loader.coalesce_caption_metadata("caption-metadata")
webshart.upload_caption_metadata(
    "caption-metadata",
    "organization/dataset-metadata",
    hf_token="hf_...",
)

The CLI provides the same operation. --shard-cache-dir lets coalescing reuse full cached shards instead of issuing one range read per sidecar:

webshart optimize-captions \
  --source organization/dataset \
  --metadata organization/dataset-metadata \
  --destination caption-metadata \
  --shard-cache-dir cache/shards \
  --push-to-hub organization/dataset-metadata

optimize-captions expects existing .tar shards and webshart indexes. To fully repackage a repository of loose media plus .txt/.json sidecars, or a legacy SimpleTuner layout containing unindexed .tar archives whose member filenames are captions, use the rolling optimize-dataset command instead:

webshart optimize-dataset \
  --source stablellama/Qwen-Image-2512_samples \
  --push-to-hub stablellama/Qwen-Image-2512_samples \
  --output-prefix webshart \
  --max-shard-size-gb 1

The target is always a Hugging Face dataset repository. It may be the same repository as the source because generated files live under --output-prefix. The input layout is detected automatically. Loose sidecars are coalesced into metadata. Legacy tar members are repacked into bounded shards and their filename stems become captions, matching SimpleTuner's filename strategy (underscores become spaces). Remote legacy inputs use aligned HTTP ranges from the saved member offset and retain only the current output shard locally.

After each shard is indexed, its sidecar captions are embedded in the JSON index and the tar, index, and .webshart-optimize-state.json are uploaded in a single commit. The state records relative positions and conversion settings, never local absolute paths. For legacy tars this includes the source archive index and tar-block member offset, so a rerun resumes within an archive after the last committed output shard. Use --max-shards N to bound each worker invocation.

For a local-only conversion, replace --push-to-hub with a local destination:

webshart optimize-dataset \
  --source /datasets/loose-pairs \
  --destination /datasets/indexed \
  --max-shards 10

Plain-text sidecars are omitted from the tar after their captions are embedded. JSON sidecars are likewise coalesced: recognized caption fields become the canonical captions value and the complete object is retained as json_metadata in the index.

Hub reads accept hf_token= and also honor HF_TOKEN. This includes gated datasets and separately hosted metadata. Local discovery recursively pairs tar and JSON indexes, preserving their relative subdirectories.

Aspect Bucketing Samples

list_shard_aspect_buckets() is file-oriented and buckets any indexed file that has width and height.

For training pipelines, prefer list_shard_sample_aspect_buckets():

loader = webshart.TarDataLoader(dataset)
buckets = loader.list_shard_sample_aspect_buckets(
    [0],
    key="geometry-tuple",
    target_pixel_area=1024**2,
)[0]["buckets"]

for bucket_key, entries in buckets.items():
    for item in entries:
        virtual_id = f"webshart://0/{item['sample_idx']}/{item['filename']}"
        image = loader.load_sample(0, item["sample_idx"])

This uses logical samples from metadata.sample_range() / get_sample_by_index() and excludes paired JSON sidecars before bucketing. Each bucket entry includes sample_idx, so callers can build stable IDs and load images directly with loader.load_sample(shard_idx, sample_idx).

Parquet Payloads With Custom Metadata

The Parquet API is available starting with webshart 0.7.0. Existing tar APIs keep their behavior. Use ParquetDataLoader, not TarDataLoader or BucketDataLoader, for Parquet sources:

dataset = webshart.discover_dataset(
    "InternVL-U/ScaleEdit-12M",
    format="parquet",
    metadata="webshart/ScaleEdit-12M-metadata",
)
loader = webshart.ParquetDataLoader(
    dataset, image_columns=["source_image", "edited_image"]
)
metadata = loader.get_metadata(0, 0)
sample = loader.load_sample(0, 0)
edited_bytes = sample.data
source_bytes = sample.payloads["source_image"]
instruction = sample.captions

Local directories work too. Without separate metadata, specify the source column names with image_column, caption_column, width_column, and height_column. discover_parquet_dataset(...) is equivalent to discovery with format="parquet". Hub discovery requires webshart[hub]; payload decoding is native Rust and does not require PyArrow, datasets, or hf_xet. Hub payload ranges reuse webshart's native Xet transport, with HTTP Range for non-Xet files.

Captured images on S3

Native S3 access and batched row-group reads require webshart 0.8.0 or later.

S3-compatible Parquet sources use native Rust AWS signing, connection pooling, ETag-pinned range reads, and streamed shard downloads. No Python S3 transfer library is required. Supply an endpoint without the bucket name:

dataset = webshart.discover_parquet_dataset(
    "s3://training-images",
    manifest="combined/dataset.json",
    s3_options={"endpoint_url": "https://storage.example", "max_connections": 32},
)
dataset.enable_shard_cache("cache/parquet", cache_limit_gb=25, parallel_downloads=4)
loader = webshart.ParquetDataLoader(dataset)
samples = loader.load_samples([(0, 12), (0, 3), (0, 12)], parallel_reads=4)

Omit manifest to list Parquet objects below the source prefix. A capture descriptor has format: "parquet-image-capture", version: 1 or 2, and a files_manifest pointing to source-relative JSONL entries containing key, size_bytes, and num_rows. An optional manifest_sha256 verifies the JSONL contents. Discovery uses manifest sizes instead of sending one HEAD per part. Object size and row counts are checked when a part is opened.

Credentials can come from AWS environment variables, web identity, or container and instance roles. Explicit aws_access_key_id, aws_secret_access_key, aws_session_token, and region_name are also accepted in s3_options. allow_http=True is available for local S3 test servers. Shared AWS profile files are not read by this transport.

Without a shard cache, only selected Parquet ranges are fetched. The shard cache streams whole objects to disk and reuses webshart's download locks and eviction. Cache entries are scoped by endpoint, bucket, object key, and ETag. Metadata preparation coalesces selected column chunks into a bounded 64 MiB cache and leaves image columns unread. Whole-shard downloads begin on payload access. Network I/O and Parquet decoding release the GIL. Batch requests share each selected row group, use bounded native concurrency, and preserve input order and duplicates.

loader.get_metadata(shard) returns logical sample metadata; loader.list_samples_in_shard(shard) and loader.list_shard_sample_aspect_buckets([shard]) omit failed capture rows. Physical row indices stay unchanged. dataset.fingerprint identifies the source configuration and catalog snapshot for downstream metadata caches. Captured bytes are read directly, without recompression or fetching the original URL. Applications decode the returned bytes; JPEG XL requires a decoder such as pillow-jxl-plugin.

Online images from URL columns

URL-only Parquet datasets such as Recap-DataComp-1B can use the same metadata overlays without storing a second copy of the images:

dataset = webshart.discover_parquet_dataset(
    "UCSC-VLAA/Recap-DataComp-1B",
    subfolder="data/train_data",
    image_url_column="url",
    caption_column="re_caption",
)
loader = webshart.ParquetDataLoader(
    dataset,
    fetch_external_urls=True,
    url_options=webshart.URLImageOptions(timeout=10, retries=1, max_bytes=16 * 1024**2),
    on_url_error="return",
)
sample = loader.load_sample(0, 0)
if sample.error is None and sample.data is not None:
    image_bytes, caption = sample.data, sample.captions

Discovery and get_metadata() never fetch external images. Without fetch_external_urls=True, URL samples have data=None. With it enabled, Rust fetches HTTP(S) bytes on demand using a reusable connection pool, bounded response sizes, a timeout covering each request and its body, and retries for transient failures. Hub credentials are never passed to image hosts. Image bytes are neither persisted nor uploaded; repeated loads fetch the URL again. The loader returns raw bytes and does not decode or validate image contents.

Failures raise URLImageError by default. on_url_error="return" yields data=None and an error string instead, preserving the physical row identity and iteration order. A missing URL is an invalid_url failure. payloads retains the original URL, and original captions and hash columns remain in metadata. An existing sha256 is not assumed to hash the currently served image bytes.

URL-source metadata reads use a bounded native Parquet cache to avoid repeated small network reads. url_metadata_cache_bytes defaults to 128 MiB per cached source shard; larger files use the existing page reader. Set it to zero to disable this cache. This caches source Parquet bytes, never fetched image bytes. The geometry scanner projects only the URL column and does not fill this cache.

Scanning image dimensions and availability

Install webshart[scan] to build metadata overlays. The Rust scanner fetches images concurrently, reads dimensions from their headers, computes downloaded byte hashes, and by default discards image bytes. Python writes checkpointed Parquet metadata. Header inspection is not a full pixel decode or image-quality check.

Start with a bounded pilot and pin the source revision for repeatable resumes:

webshart scan-url-metadata \
  --source UCSC-VLAA/Recap-DataComp-1B \
  --revision 457c44d98651bcfdfb3cc8695f5e60a0d2705e78 \
  --subfolder data/train_data \
  --url-column url --caption-column re_caption \
  --destination ./recap-metadata \
  --workers 128 --batch-size 1000 \
  --timeout 10 --retries 1 --max-bytes 16777216 \
  --max-samples 1000

--workers bounds simultaneous native requests, not Python threads. Peak image buffer memory scales with concurrency times --max-bytes; metadata batches are bounded by --batch-size. Checkpoints are separate per source shard. Re-running with the same source and fetch options resumes after committed batches; --max-samples limits new attempts in each invocation. Worker count and batch size may change. An interrupted batch is retried. Previously recorded failures are retained; use a fresh destination to reassess them. Do not run two writers against the same destination. For distributed scans, use disjoint --start-shard/--stop-shard ranges and separate output directories.

Add --capture-images to also write image-bearing Parquet shards under data/. Each row contains image: {bytes, path}, captions, url, physical source_shard and source_row, and the scan metadata. Failed probes retain null images and their failure status. Filter url_status == "ok" for training. Rust stages successful downloads while Python writes bounded Arrow row groups; images are fetched only once per successful batch.

Optional --recompress-images (SDK: recompress_images=True) losslessly recompresses captured JPEGs to JPEG XL. Install the reference cjxl and djxl tools on PATH; missing tools fail the scan before downloading a batch. Rust invokes the codecs without a shell, limits each codec to one thread, reconstructs the JPEG, and compares the reconstructed file byte for byte with the download. Only smaller, verified results replace originals. Other formats, codec failures, and conversions that do not save space retain the original bytes. This is JPEG bitstream recompression, not lossy pixel re-encoding.

--recompress-workers (SDK: recompress_workers, default 2) separately bounds encoding concurrency per scan invocation. Codec calls have a 60-second timeout each. Rust performs staging, hashing, verification, and cleanup with the Python GIL released. The codec implementation is supplied by the external libjxl tools. Captured rows include stored_format (jxl or original), stored_bytes, stored_sha256, and recompression_status; the existing download_* fields always describe the original response. Failed downloads have null storage fields. JXL bytes require a compatible training decoder or JPEG reconstruction with djxl; webshart returns the stored bytes unchanged. Enabling recompression requires a fresh scan destination so a resume cannot silently mix policies.

To upload each completed shard and discard its local image copy, add:

--capture-images --upload-repo owner/dataset --no-local-images

The destination repository must already exist. Uploads use huggingface_hub authentication and endpoint configuration; --upload-endpoint overrides only the upload endpoint. --upload-repo-type model supports repositories exposed through the model API. The SDK equivalents are capture_images=True, upload_repo="owner/dataset", keep_local_images=False, and optional upload_endpoint, upload_token, and upload_repo_type arguments to scan_url_metadata. Explicit upload credentials do not change source reads.

This mode needs temporary disk space for one download batch and its Parquet shard per process (bounded by batch size times the maximum image size, plus encoding overhead). It retains local metadata and checkpoints. Images are removed only after a successful Hub commit and a durable local checkpoint. A failed upload retains the completed batch for retry without refetching. Resuming trusts recorded Hub commit receipts; keep the uploaded files intact. The destination data/ files are self-contained image/caption Parquet shards; the metadata overlay still refers to the original URL source. Local captured shards can be loaded with discover_parquet_dataset("./recap-metadata/data"). For a remote captured repository, use subfolder="data" and optionally endpoint and repo_type="model" in discover_parquet_dataset. Use a fresh scan destination when enabling capture on an existing metadata-only scan: discarded image bytes must be downloaded again. Use separate upload repositories for different sources or revisions, and disjoint shard ranges for concurrent writers to one repository.

For a separate uploader, give each producer a distinct child scan directory and matching staging directory, then add --defer-uploads --no-local-images with --upload-repo. For example, metadata in ./scans/000000 can stage bytes in ./buffer/000000 using --staging-directory. Downloads, native recompression, and image Parquet writing continue without waiting for Hub uploads. --min-staging-free-bytes pauses producers when the staging filesystem is low on free space. Keep staging directories exclusive to their corresponding scans.

A separate process can periodically run:

webshart upload-capture-queue \
  --destination ./scans --staging-directory ./buffer \
  --upload-repo owner/dataset --min-bytes 1099511627776

The uploader consumes completed child scans once at least 1 TiB is pending. It retains individual Parquet files and uploads groups of at most 64 files per commit, with four transfer threads by default. Only one uploader may operate on a queue. The SDK equivalent is upload_capture_queue; its byte threshold, commit size, and transfer concurrency are configurable. Run with --flush after all producers finish to upload the final remainder. If resuming an interrupted batch, repeat until the returned pending_bytes is zero. Capture completion alone is not proof of upload completion.

Active batch state and commit receipts remain with the scan metadata. A small receipt marker is committed alongside each remote file group, allowing retries to recognize a successful commit even if its acknowledgment was lost. Staged files are removed after the receipt and checkpoints are saved. Preserve both metadata and staging across restarts; missing pending image files cause an error rather than silently skipping rows. Deferred mode is part of scan identity; existing immediate-upload scans require an explicit checkpoint migration or a fresh destination before changing modes.

The resulting webshart-parquet.json and metadata/ files can be used locally or hosted together in a metadata-only Hub repository. Keep scan-state.json, checkpoints/, and parts/ locally for resuming. Each completed invocation publishes an overlay covering the rows attempted so far, including partial shards; unscanned rows retain their original metadata.

dataset = webshart.discover_parquet_dataset(
    "UCSC-VLAA/Recap-DataComp-1B",
    metadata="./recap-metadata",
)
loader = webshart.ParquetDataLoader(dataset, fetch_external_urls=True)
metadata = loader.get_metadata(0, 0)
if metadata.get("url_status") == "ok":
    image = loader.load_sample(0, 0)

Overlay rows include width, height, url_status, url_http_status, download_bytes, and download_sha256. scan-summary.json separates successful geometry reads, HTTP status codes, timeouts, network errors, size limits and invalid images. These are observations from one scan: access denial, temporary failures, and unsupported formats are not proof of permanent link rot. A prefix pilot is not a representative estimate of availability across the whole dataset. Image URLs can change even when the Parquet source revision is pinned, and a successful scan does not guarantee a later training request will succeed.

Metadata manifest format

Pass a local metadata directory or Hub dataset repository containing webshart-parquet.json. The source files stay unchanged. For example:

{
  "format": "webshart-parquet",
  "version": 1,
  "source": {"repo_id": "owner/source", "revision": "<40-character commit SHA>"},
  "columns": {"image": "image", "captions": "instruction", "width": "width", "height": "height"},
  "metadata_mode": "overlay",
  "shards": [{
    "path": "data/part-000.parquet",
    "num_rows": 2,
    "metadata_path": "custom/part-000.json"
  }]
}

For URL sources, set "image_url": "url" in columns instead of "image". This declares the payload column but does not enable network loading; the loader still requires fetch_external_urls=True. Overlays cannot replace payload URLs.

The JSON metadata file is a list of row objects:

[
  {"source_row": 1, "captions": [{"scene": "sunset"}, "warm lighting"], "rating": 5},
  {"source_row": 0, "captions": {"scene": "forest", "lighting": "soft"}}
]

Alternatively, metadata_path can name a Parquet file with a source_row integer column and arbitrary custom columns. Row order and source IDs do not matter: the join uses the source shard path plus zero-based physical row. Optional source_path values must match the manifest. Duplicates, invalid row positions, unsafe paths, mismatched file sizes/row counts, and incorrect optional metadata_sha256 checksums fail explicitly. size, when present, is the source Parquet file size in bytes. Remote manifests must pin their source commit; metadata_revision can additionally pin the metadata repository commit.

overlay merges custom values over source non-binary columns and permits sparse overrides. Custom captions, width, and height override configured aliases. replace reads metadata only from the companion file and requires every row to be present. It avoids reading upstream metadata columns. Payload columns cannot be overridden. Metadata is cached per shard with a bounded LRU (default: one shard; adjust metadata_cache_size). Recreate the dataset after changing files. Companion files up to 64 MiB are buffered while decoding so many small Parquet pages do not cause repeated network round trips; image payload files are not.

Parquet structs, lists, and maps retain their native Python representations. For mixed strings/objects that Parquet cannot represent in one homogeneous column, store JSON text and declare "json_columns": ["captions"] in the manifest (or discovery options). Only explicitly declared columns are parsed; ordinary caption strings are not guessed or stringified. Decimal values remain decimal.Decimal; date/time values use their Parquet integer units.

Streaming and access costs

loader.iter_shard(shard_index, start=0, stop=None) streams selected image columns page by page, reusing dictionaries. load_sample(shard_index, row_index) skips preceding row groups but scans earlier rows within the selected group. Unlike indexed tar, compressed Parquet does not provide constant-cost per-image byte offsets. Prefer sequential iteration for large row groups. Memory scales with selected columns' decoded pages/dictionaries, not an entire shard; oversized dictionaries can still require hundreds of MiB. The max_page_bytes discovery option limits individual compressed page/footer reads (default 512 MiB), not total or decompressed memory.

Binary image columns and Hugging Face {bytes, path} image structs are supported. Missing bytes produce sample.data is None; original structs remain in sample.payloads. External URLs/path-only images are never fetched automatically. Keep URL/hash metadata and apply your own retrieval policy. The loader does not rewrite or download whole source shards, and does not expose tar-only byte-offset, bucket, shard-cache, or field-rewrite APIs.

Why is indexed tar fast?

Problem: Standard tar files require sequential reading. To get file #10,000, you must read through files #1-9,999 first.

Solution: The indexed format stores byte offsets and sample metadata in a separate JSON file, enabling:

  • HTTP range requests for any file
  • True random access over network
  • Parallel reads from multiple shards
  • Large scale, aspect-bucketed datasets
  • No wasted bandwidth

The Rust implementation provides:

  • Real parallelism (no Python GIL)
  • Zero-copy operations where possible
  • Efficient HTTP connection pooling
  • Optimized tokio async runtime
  • Optional local caching for metadata and shards
  • Fast aspect bucketing for image data

Datasets Using This Format

I discovered after creating this library that cheesechaser is the origin of the indexed tar format, which webshart has formalised and extended to include aspect bucketing support.

Requirements

  • Python 3.12+
  • Linux/macOS/Windows

Roadmap

  • image decoding is currently not handled by this library, but it will be added with zero-copy.
  • more informative API for caching and other Rust implementation details
  • multi-gpu/multi-node friendly dataloader

Projects using webshart

  • CaptionFlow uses this library to solve memory use and seek performance issues typical to webdatasets

License

MIT

Metadata

Release files for webshart 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for webshart 0.8.0
File Size Uploaded Provenance Actions
webshart-0.8.0.tar.gz 239.0 kB Without provenance Details for webshart-0.8.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for webshart 0.8.0
File Actions
webshart-0.8.0-cp312-abi3-win_amd64.whl CPython 3.12 abi3 Windows x86-64 Details for webshart-0.8.0-cp312-abi3-win_amd64.whl
webshart-0.8.0-cp312-abi3-manylinux_2_39_x86_64.whl CPython 3.12 abi3 Linux glibc 2.39+ x86-64 Details for webshart-0.8.0-cp312-abi3-manylinux_2_39_x86_64.whl
webshart-0.8.0-cp312-abi3-manylinux_2_39_aarch64.whl CPython 3.12 abi3 Linux glibc 2.39+ ARM64 Details for webshart-0.8.0-cp312-abi3-manylinux_2_39_aarch64.whl
webshart-0.8.0-cp312-abi3-manylinux_2_35_x86_64.whl CPython 3.12 abi3 Linux glibc 2.35+ x86-64 Details for webshart-0.8.0-cp312-abi3-manylinux_2_35_x86_64.whl
webshart-0.8.0-cp312-abi3-macosx_11_0_arm64.whl CPython 3.12 abi3 macOS 11.0+ ARM64 Details for webshart-0.8.0-cp312-abi3-macosx_11_0_arm64.whl

Total release size: 36.8 MB

Release files / webshart-0.8.0.tar.gz

Download URL webshart-0.8.0.tar.gz
Size 239.0 kB
Tags Source
SHA-256 checksum
How to use checksums
b79337173671774df42c10db01818ae899fed0bcbca8dea5bc46b485b9dc45a3
BLAKE2b-256 checksum
How to use checksums
4a56bf039b3adfec1ed51bc4ceab82d711e55f78fce6da2b76c8807137d19004
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Release files / webshart-0.8.0-cp312-abi3-win_amd64.whl

Download URL webshart-0.8.0-cp312-abi3-win_amd64.whl
Size 7.2 MB
Tags CPython 3.12 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
47c70742bf1723a5e60fe35d0fbe1f5e84585b4f70265261c25362355461058a
BLAKE2b-256 checksum
How to use checksums
2edd72fdb3533d92dfe9d94630bcc6af073578e14dea75ec1da2b3f4cbe0db16
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Release files / webshart-0.8.0-cp312-abi3-manylinux_2_39_x86_64.whl

Download URL webshart-0.8.0-cp312-abi3-manylinux_2_39_x86_64.whl
Size 7.6 MB
Tags CPython 3.12 Linux glibc 2.39+ x86-64 abi3
SHA-256 checksum
How to use checksums
000bf1349cdc21130d6274a1c0798b1d58d84170323dd25885966d05b840583e
BLAKE2b-256 checksum
How to use checksums
ab6e8d90f89cf701a0f39c1f63e966789ccd509486f42ad240b90255e672d81a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Release files / webshart-0.8.0-cp312-abi3-manylinux_2_39_aarch64.whl

Download URL webshart-0.8.0-cp312-abi3-manylinux_2_39_aarch64.whl
Size 7.2 MB
Tags CPython 3.12 Linux glibc 2.39+ ARM64 abi3
SHA-256 checksum
How to use checksums
0414a73a452c7d555bcaf77150cb339e150ab28bf197b047887df2bf4147c6ce
BLAKE2b-256 checksum
How to use checksums
3b31442adfc02cdf90dcb85c25611cb0f7306ae8fc120fdc8385742750b6a478
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Release files / webshart-0.8.0-cp312-abi3-manylinux_2_35_x86_64.whl

Download URL webshart-0.8.0-cp312-abi3-manylinux_2_35_x86_64.whl
Size 7.6 MB
Tags CPython 3.12 Linux glibc 2.35+ x86-64 abi3
SHA-256 checksum
How to use checksums
588cf6d89b8a4d67907ce111cb9b59e843e0a626d956e80c2490de35b4cd79f6
BLAKE2b-256 checksum
How to use checksums
fc590a32d4a49d158766f4865d43eb820c42b9f07438fce3e62c2d8ea4bd1eed
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Release files / webshart-0.8.0-cp312-abi3-macosx_11_0_arm64.whl

Download URL webshart-0.8.0-cp312-abi3-macosx_11_0_arm64.whl
Size 7.0 MB
Tags CPython 3.12 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
02e9809b1b833be68cf66f0dd5187374030b811f48ec67b42a8d06e4a2da5312
BLAKE2b-256 checksum
How to use checksums
9468197e5ebb22d5f366b803be8468205e5fe77521c6c6998d460bf886170b6e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.12.8

Security for webshart 0.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page