Fast dataloader and conversion utility for webdataset tar shards. Rust core with Python bindings.
Built for streaming large video and image datasets, but handles any byte data.
Install
pip install webshart
What is this?
Webshart is a fast reader for webdataset tar files with separate JSON index files. This format enables random access to any file in the dataset without downloading the entire archive.
The indexed format provides massive performance benefits:
- Random access: Jump to any file instantly
- Selective downloads: Only fetch the files you need
- True parallelism: Read from multiple shards simultaneously
- Cloud-optimized: Works efficiently with HTTP range requests
- Aspect bucketing: Optionally include image geometry hints
width,heightandaspectfor the ability to bucket images by shape - Logical sample APIs: Treat
image.ext+image.jsonpairs as one sample while still allowing raw file access - Caption metadata: Store captions under
captionsas strings, native JSON objects, or mixed lists of both - Custom DataLoader: Includes state dict methods on the DataLoader so that you can resume training deterministically
- Rate-limit friendly: Local caching allows high-frequency random seeking without encountering storage provider rate limits
- Instant start-up with pre-sorted aspect buckets
Growing ecosystem: While not all datasets use this format yet, you can easily create indices for any tar-based dataset (see below).
Quick Start
import webshart
# Find your dataset
dataset = webshart.discover_dataset(
source="laion/conceptual-captions-12m-webdataset",
# we're able to upload metadata separately so that we reduce load on huggingface infra.
metadata="webshart/conceptual-captions-12m-webdataset-metadata",
)
print(f"Found {dataset.num_shards} shards")
loader = webshart.TarDataLoader(dataset)
# File-oriented access is still available.
files = dataset.list_files_in_shard(0)
# Sample-oriented access skips paired JSON sidecars.
samples = dataset.list_samples_in_shard(0)
entry = loader.load_sample(0, 0)
print(entry.path, entry.captions, entry.json_metadata)
Paired datasets
Two datasets can remain independently loadable while also exposing an opt-in join by logical sample key. This works especially well for preference, reference, and slider-training data stored in two subfolders of one repository:
paired = webshart.discover_paired_dataset(
"webshart/suno-various-94k",
left_subfolder="original",
right_subfolder="covers",
)
print(paired.num_pairs)
print(paired.get_pair(0))
loader = webshart.PairedTarDataLoader(paired)
sample = loader.load_pair(0)
print(sample.key, sample.left, sample.right)
The normal contract is unchanged: calling discover_dataset(..., subfolder="original") or subfolder="covers" returns a standalone dataset.
Pair indexing is lazy, preserves left-dataset order, and validates identical key
sets by default. Pass strict=False to use only the intersection and inspect
unmatched_left / unmatched_right.
max_file_size is a visibility limit for loader APIs. Files larger than the
configured limit are omitted from iteration, batches, direct sample loading,
and aspect buckets instead of being returned with empty data. Direct
load_sample() calls return None for an oversized sample. The loader's
list_samples_in_shard() returns dictionaries containing sample_idx and
filename, so filtered listings retain the stable index required by
load_sample().
Native Hugging Face Xet Downloads
Hugging Face files stored in Xet are read directly by webshart's Rust transport.
Neither hf_xet nor huggingface_hub is needed for dataset reads. Non-Xet files
and other HTTP servers continue to use ordinary HTTP downloads and byte ranges.
The transport supports full-shard caching, individual sample/sidecar reads, and batched range reads. It reconstructs files using concurrent 16 MiB windows, decodes uncompressed, LZ4, and byte-grouped LZ4 chunks, and handles CAS v2 multi-range responses and the documented v1 compatibility path. Buffers and request concurrency are bounded rather than proportional to the entire shard. Full-shard downloads validate the size and Hub-provided SHA-256 before entering the cache; partial reads validate ranges, chunk sizes, and reconstruction lengths. Transient requests are retried, expired credentials/transfer URLs are refreshed, and native Xet errors are reported rather than silently bypassing verification.
dataset = webshart.discover_dataset("webshart/pseudo-camera-10k-structured", subfolder="data")
dataset.enable_shard_cache("cache/shards", cache_limit_gb=25, parallel_downloads=4)
loader = webshart.TarDataLoader(dataset)
loader.prepare_shards_ahead(4)
sample = loader.load_sample(0, 0)
parallel_downloads limits concurrent whole shards in each cache; prefetch must
be scheduled explicitly. Xet independently permits up to eight simultaneous
chunk transfers per process, including multiple transfers within one shard.
Set WEBSHART_XET_CONCURRENCY=1..64 before the first read to change that limit.
load_sample() releases the Python GIL during I/O. Without a shard cache, only
the requested sample ranges are reconstructed, not the entire archive.
Set WEBSHART_DISABLE_XET=1 to explicitly use the ordinary HTTP path instead.
HF_ENDPOINT selects the Hub origin recognized by the Xet transport for custom
resolve URLs; it does not change repository discovery's Hub endpoint.
Implemented from the public download protocol,
authentication specification, and
xorb format. Run the opt-in live test with
cargo test --lib xet::tests::live_public_hub_shard -- --ignored --nocapture;
it downloads a 287 MiB public shard, verifies its SHA-256, and compares partial
reads with independent HTTP ranges.
WEBSHART_TEST_LIVE_XET=1 .venv/bin/python -m pytest -q tests/test_xet_live.py
also exercises the Python SDK, cache, batched reads, and ordinary HTTP opt-out
with both Hugging Face Python libraries blocked from import.
Common Patterns
For real-world, working examples:
- Use as a DataLoader
- Retrieve data subset/range
- Get dataset statistics without downloading
- List aspect buckets
- Write captions into metadata
Creating Indices for / Converting Existing Datasets
Any tar-based webdataset can benefit from indexing! Webshart includes tools to generate indices:
A command-line tool that auto-discovers tars to process:
% webshart extract-metadata \
--source laion/conceptual-captions-12m-webdataset \
--destination laion_output/ \
--checkpoint-dir ./laion_output/checkpoints \
--max-workers 2 \
--include-image-geometry
Or, if you prefer/require direct-integration to an existing Python application, use the API
Uploading Indices to HuggingFace
Once you've generated indices, share them with the community:
# Upload all JSON files to your dataset
huggingface-cli upload --repo-type=dataset \
username/dataset-name \
./indices/ \
--include "*.json" \
--path-in-repo "indices/"
Or if you want to contribute to an existing dataset you don't own:
- Create a community dataset with indices:
username/original-dataset-indices - Upload the JSON files there
- Open a discussion on the original dataset suggesting they add the indices
Creating New Indexed Datasets
If you're creating a new dataset, generate indices during creation:
{
"files": {
"image_0001.webp": {"offset": 512, "length": 102400},
"image_0002.webp": {"offset": 102912, "length": 98304},
...
}
}
The JSON index should have the same name as the tar file (e.g., shard_0000.tar → shard_0000.json).
Caption layouts and sidecars
Webshart recognizes both JSON metadata sidecars and plain-text caption sidecars:
sample_0001.webp
sample_0001.json
sample_0002.webp
sample_0002.txt
Paired .json and .txt members are excluded from logical sample indexes. You
can inspect the layout from shard metadata without downloading tar members:
layout = dataset.probe_caption_layout(max_shards=16)
print(layout["layout"]) # embedded, json_sidecar, txt_sidecar, mixed, or none
When metadata is extracted or loaded, sidecars are attached to their paired sample entries:
{
"files": {
"sample_0001.webp": {
"offset": 512,
"length": 102400,
"width": 1024,
"height": 1024,
"aspect": 1.0,
"json_path": "sample_0001.json",
"json_offset": 103424,
"json_length": 128,
"captions": "a product photo on a white background",
"json_metadata": {
"caption": "a product photo on a white background"
}
},
"sample_0001.json": {
"offset": 103424,
"length": 128
}
}
}
Use file-oriented APIs when you want every archive member, including sidecars:
dataset.list_files_in_shard(0)
reader = dataset.open_shard(0)
raw_file_bytes = reader.read_file(0)
Use sample-oriented APIs when you want training samples:
dataset.list_samples_in_shard(0)
dataset.get_shard_sample_count(0)
reader = dataset.open_shard(0)
image_bytes = reader.read_sample(0)
json_bytes = reader.read_sample_json(0)
entry = loader.load_sample(0, 0)
print(entry.path)
print(entry.captions)
print(entry.json_data)
# Direct caption lookup also handles paired .txt sidecars.
caption = loader.load_caption(0, 0)
Captions are canonicalized to the plural captions metadata key. The value may be
a string, a native JSON object, a list containing either, or absent. Nested
objects, arrays, numbers, and booleans inside caption objects retain their JSON
types. entry.captions returns the complete value; entry.caption and
loader.load_caption() return the first string or object.
The optimizer and caption coalescer recognize JSON objects and mixed caption
lists in .txt sidecars. Plain text remains a string. JSON sidecars support
native values under their caption fields and standalone caption objects.
webshart.write_captions_to_metadata(
"shard_0000.json",
{"sample_0001.webp": {"description": "a café", "elements": [{"bbox": [1, 2, 30, 40]}]}},
)
webshart.write_captions_to_metadata(
"shard_0000.json",
{
"sample_0001.webp": "a short caption",
"sample_0002": ["caption one", "caption two"],
},
)
The writer updates existing webshart metadata JSON in place, removes old singular caption keys from updated samples, and leaves paired .json sidecar entries untouched.
Renaming metadata fields
Permanently rename a field across a dataset's JSON indexes:
dataset = webshart.discover_dataset("/path/to/dataset")
updated = dataset.field_rename("captions", "v1_captions")
loader = webshart.TarDataLoader(dataset)
entry = loader.load_sample(0, 0)
print(entry.metadata["v1_captions"])
field_rename(old_name, new_name, *, overwrite=False, destination=None) returns
the number of file entries renamed. It moves top-level keys within each index's
files entries, preserving values, custom fields, and dict/list index layouts.
It processes one shard's metadata at a time without reading or rewriting tar
payloads. Nested json_metadata and sidecar contents are left intact.
Missing fields are skipped; renaming a field to itself returns zero. Existing
destination fields raise ValueError unless overwrite=True. Structural index
fields such as offsets, lengths, and paths cannot be renamed. All indexes are
validated before replacements begin, and each file replacement is atomic;
an I/O failure during replacement can still leave earlier shards updated.
Local indexes are rewritten in place and their loaded/disk metadata caches are
invalidated. Recreate existing loaders after renaming. Custom names are available
through entry.metadata and loader.get_metadata(); entry.captions continues
to represent the canonical caption field and may still read unchanged sidecars.
For remote indexes, pass destination="./renamed-metadata" to write a local
export, then upload those JSON files separately. This also works for local
datasets when an export is preferred. The dataset uses the exported indexes
afterward, and tar files stay at their original locations.
Coalescing caption metadata
To avoid repeated .txt range reads, fold all sidecar captions into standard
webshart metadata files. If metadata caching is enabled, omitting the destination
persists the enriched indexes in webshart's cache:
dataset.enable_metadata_cache("cache/metadata", init_shard_count=0)
loader = webshart.TarDataLoader(dataset, load_file_data=False)
loader.coalesce_caption_metadata()
# Or create a portable export tree for copying or upload.
loader.coalesce_caption_metadata("caption-metadata")
webshart.upload_caption_metadata(
"caption-metadata",
"organization/dataset-metadata",
hf_token="hf_...",
)
The CLI provides the same operation. --shard-cache-dir lets coalescing reuse
full cached shards instead of issuing one range read per sidecar:
webshart optimize-captions \
--source organization/dataset \
--metadata organization/dataset-metadata \
--destination caption-metadata \
--shard-cache-dir cache/shards \
--push-to-hub organization/dataset-metadata
optimize-captions expects existing .tar shards and webshart indexes. To
fully repackage a repository of loose media plus .txt/.json sidecars, or a
legacy SimpleTuner layout containing unindexed .tar archives whose member
filenames are captions, use the rolling optimize-dataset command instead:
webshart optimize-dataset \
--source stablellama/Qwen-Image-2512_samples \
--push-to-hub stablellama/Qwen-Image-2512_samples \
--output-prefix webshart \
--max-shard-size-gb 1
The target is always a Hugging Face dataset repository. It may be the same
repository as the source because generated files live under --output-prefix.
The input layout is detected automatically. Loose sidecars are coalesced into
metadata. Legacy tar members are repacked into bounded shards and their
filename stems become captions, matching SimpleTuner's filename strategy
(underscores become spaces). Remote legacy inputs use aligned HTTP ranges from
the saved member offset and retain only the current output shard locally.
After each shard is indexed, its sidecar captions are embedded in the JSON
index and the tar, index, and .webshart-optimize-state.json are uploaded in a
single commit. The state records relative positions and conversion settings,
never local absolute paths. For legacy tars this includes the source archive
index and tar-block member offset, so a rerun resumes within an archive after
the last committed output shard. Use --max-shards N to bound each worker
invocation.
For a local-only conversion, replace --push-to-hub with a local destination:
webshart optimize-dataset \
--source /datasets/loose-pairs \
--destination /datasets/indexed \
--max-shards 10
Plain-text sidecars are omitted from the tar after their captions are embedded.
JSON sidecars are likewise coalesced: recognized caption fields become the
canonical captions value and the complete object is retained as
json_metadata in the index.
Hub reads accept hf_token= and also honor HF_TOKEN. This includes gated
datasets and separately hosted metadata. Local discovery recursively pairs tar
and JSON indexes, preserving their relative subdirectories.
Aspect Bucketing Samples
list_shard_aspect_buckets() is file-oriented and buckets any indexed file that has width and height.
For training pipelines, prefer list_shard_sample_aspect_buckets():
loader = webshart.TarDataLoader(dataset)
buckets = loader.list_shard_sample_aspect_buckets(
[0],
key="geometry-tuple",
target_pixel_area=1024**2,
)[0]["buckets"]
for bucket_key, entries in buckets.items():
for item in entries:
virtual_id = f"webshart://0/{item['sample_idx']}/{item['filename']}"
image = loader.load_sample(0, item["sample_idx"])
This uses logical samples from metadata.sample_range() / get_sample_by_index() and excludes paired JSON sidecars before bucketing. Each bucket entry includes sample_idx, so callers can build stable IDs and load images directly with loader.load_sample(shard_idx, sample_idx).
Parquet Payloads With Custom Metadata
The Parquet API is available starting with webshart 0.7.0. Existing tar APIs
keep their behavior. Use ParquetDataLoader, not
TarDataLoader or BucketDataLoader, for Parquet sources:
dataset = webshart.discover_dataset(
"InternVL-U/ScaleEdit-12M",
format="parquet",
metadata="webshart/ScaleEdit-12M-metadata",
)
loader = webshart.ParquetDataLoader(
dataset, image_columns=["source_image", "edited_image"]
)
metadata = loader.get_metadata(0, 0)
sample = loader.load_sample(0, 0)
edited_bytes = sample.data
source_bytes = sample.payloads["source_image"]
instruction = sample.captions
Local directories work too. Without separate metadata, specify the source column
names with image_column, caption_column, width_column, and height_column.
discover_parquet_dataset(...) is equivalent to discovery with format="parquet".
Hub discovery requires webshart[hub]; payload decoding is native Rust and does
not require PyArrow, datasets, or hf_xet. Hub payload ranges reuse webshart's
native Xet transport, with HTTP Range for non-Xet files.
Counting Parquet image rows
len(dataset) counts physical rows. Call
dataset.count_rows(exclude_null_images=True) when you need to count rows
with embedded image bytes. Pass shard_index to limit either count to one
shard. A zero-length byte string is present, and the count leaves physical row
indices unchanged.
The image count uses footer statistics where available. For row groups without
null statistics, the reader scans the selected image column in bounded batches.
The count applies to embedded bytes, including the bytes field in an image
struct.
Projected metadata reads
Read selected physical rows and columns without materializing every row's metadata in the shard:
rows = dataset.read_metadata_rows(0, [12, 3, 12], fields=["width", "height"])
samples = loader.load_samples([(0, 12), (0, 3), (0, 12)], metadata_fields=[])
for entry in loader.iter_samples_in_shard(0, metadata_fields=["width", "height"], batch_size=256):
print(entry["sample_idx"], entry["width"], entry["height"])
Requested order and duplicates are preserved. Every metadata result includes
source_path and source_row; fields=[] returns only that identity.
captions, width, and height resolve configured column aliases and custom
overrides. Payload columns are excluded. metadata_fields=[] on load_samples
loads image payloads without captions or other optional metadata. Omitting
fields or metadata_fields returns all non-payload metadata.
Projected sample listings still read capture status fields and omit failed
captures. Aspect buckets accept metadata_fields too and add the dimensions
needed for bucketing. list_samples_in_shard materializes the resulting list;
iter_samples_in_shard bounds each projected read by batch_size.
Capture manifests and custom metadata overlays use temporary SQLite indexes.
Discovery streams and verifies the capture JSONL manifest without retaining a
Python dictionary per shard or sending a HEAD request per object. For capture
manifests, dataset.physical_row_count and len(dataset) return the published
physical row count without opening payloads; this includes failed fetches.
physical_row_count is None for other discovery modes.
Custom overlays are fully validated when their index is built, including sparse
row identity, duplicate rows, replacement coverage, and optional checksums.
Only requested source columns are decoded by the native reader. Index and reader
counts follow metadata_cache_size; large selected dictionaries can still cost
substantial memory, and JSON companions retain their 16 MiB limit.
Cached payload readers also follow the shard cache's byte budget, while allowing
one oversized shard to be opened.
For a persisted dataset view, save dataset.source_revision_for_shard(index)
alongside each shard's accepted rows, then call
dataset.bind_source_revisions(lookup) on the reopened dataset. lookup(index)
returns the saved revision or None for an unpinned shard. New reader opens reject
changed content: local files use SHA-256, Hub files use the pinned commit, and S3
objects use ETags. Unhashed custom overlays are included in the revision. Binding
a lookup clears existing reader and metadata caches.
dataset.io_stats() reports logical S3 HEAD, range, download, and listing
operations, plus transferred range/download bytes; SDK retries are internal.
It returns an empty dictionary for other transports. These counters include
manifest I/O. Per-shard sha256 fields in a capture manifest are compared with
S3's sha256 object metadata when that metadata is present; range reads retain
their ETag checks.
Captured images on S3
Native S3 access and batched row-group reads require webshart 0.8.0 or later.
S3-compatible Parquet sources use native Rust AWS signing, connection pooling, ETag-pinned range reads, and streamed shard downloads. No Python S3 transfer library is required. Supply an endpoint without the bucket name:
dataset = webshart.discover_parquet_dataset(
"s3://training-images",
manifest="combined/dataset.json",
s3_options={"endpoint_url": "https://storage.example", "max_connections": 32},
)
dataset.enable_shard_cache("cache/parquet", cache_limit_gb=25, parallel_downloads=4)
loader = webshart.ParquetDataLoader(dataset)
samples = loader.load_samples([(0, 12), (0, 3), (0, 12)], parallel_reads=4)
Omit manifest to list Parquet objects below the source prefix. A capture
descriptor has format: "parquet-image-capture", version: 1 or 2, and a
files_manifest pointing to source-relative JSONL entries containing key,
size_bytes, and num_rows. An optional manifest_sha256 verifies the JSONL
contents. Discovery uses manifest sizes instead of sending one HEAD per part.
Object size and row counts are checked when a part is opened.
Credentials can come from AWS environment variables, web identity, or container
and instance roles. Explicit aws_access_key_id, aws_secret_access_key,
aws_session_token, and region_name are also accepted in s3_options.
allow_http=True is available for local S3 test servers. Shared AWS profile files
are not read by this transport.
Without a shard cache, only selected Parquet ranges are fetched. The shard cache streams whole objects to disk and reuses webshart's download locks and eviction. Cache entries are scoped by endpoint, bucket, object key, and ETag. Metadata preparation coalesces selected column chunks into a bounded 64 MiB cache and leaves image columns unread. Whole-shard downloads begin on payload access. Network I/O and Parquet decoding release the GIL. Batch requests share each selected row group, use bounded native concurrency, and preserve input order and duplicates.
loader.get_metadata(shard) returns logical sample metadata;
loader.list_samples_in_shard(shard) and
loader.list_shard_sample_aspect_buckets([shard]) omit failed capture rows.
Physical row indices stay unchanged. dataset.fingerprint identifies the source
configuration and catalog snapshot for downstream metadata caches. Captured bytes are read directly, without
recompression or fetching the original URL. Applications decode the returned
bytes; JPEG XL requires a decoder such as pillow-jxl-plugin.
Online images from URL columns
URL-only Parquet datasets such as Recap-DataComp-1B can use the same metadata overlays without storing a second copy of the images:
dataset = webshart.discover_parquet_dataset(
"UCSC-VLAA/Recap-DataComp-1B",
subfolder="data/train_data",
image_url_column="url",
caption_column="re_caption",
)
loader = webshart.ParquetDataLoader(
dataset,
fetch_external_urls=True,
url_options=webshart.URLImageOptions(timeout=10, retries=1, max_bytes=16 * 1024**2),
on_url_error="return",
)
sample = loader.load_sample(0, 0)
if sample.error is None and sample.data is not None:
image_bytes, caption = sample.data, sample.captions
Discovery and get_metadata() never fetch external images. Without
fetch_external_urls=True, URL samples have data=None. With it enabled,
Rust fetches HTTP(S) bytes on demand using a reusable connection pool, bounded
response sizes, a timeout covering each request and its body, and retries for
transient failures. Hub credentials are never passed to image hosts. Image
bytes are neither persisted nor uploaded; repeated loads fetch the URL again.
The loader returns raw bytes and does not decode or validate image contents.
Failures raise URLImageError by default. on_url_error="return" yields
data=None and an error string instead, preserving the physical row identity
and iteration order. A missing URL is an invalid_url failure. payloads retains
the original URL, and original captions and hash columns remain in metadata.
An existing sha256 is not assumed to hash the currently served image bytes.
URL-source metadata reads use a bounded native Parquet cache to avoid repeated
small network reads. url_metadata_cache_bytes defaults to 128 MiB per cached
source shard; larger files use the existing page reader. Set it to zero to
disable this cache. This caches source Parquet bytes, never fetched image bytes.
The geometry scanner projects only the URL column and does not fill this cache.
Scanning image dimensions and availability
Install webshart[scan] to build metadata overlays. The Rust scanner fetches
images concurrently, reads dimensions from their headers, computes downloaded
byte hashes, and by default discards image bytes. Python writes checkpointed Parquet
metadata. Header inspection is not a full pixel decode or image-quality check.
Start with a bounded pilot and pin the source revision for repeatable resumes:
webshart scan-url-metadata \
--source UCSC-VLAA/Recap-DataComp-1B \
--revision 457c44d98651bcfdfb3cc8695f5e60a0d2705e78 \
--subfolder data/train_data \
--url-column url --caption-column re_caption \
--destination ./recap-metadata \
--workers 128 --batch-size 1000 \
--timeout 10 --retries 1 --max-bytes 16777216 \
--max-samples 1000
--workers bounds simultaneous native requests, not Python threads. Peak image
buffer memory scales with concurrency times --max-bytes; metadata batches are
bounded by --batch-size. Checkpoints are separate per source shard. Re-running
with the same source and fetch options resumes after committed batches;
--max-samples limits new attempts in each invocation. Worker count and batch
size may change. An interrupted batch is retried. Previously recorded failures
are retained; use a fresh destination to reassess them. Do not run two writers
against the same destination. For distributed scans, use disjoint
--start-shard/--stop-shard ranges and separate output directories.
Add --capture-images to also write image-bearing Parquet shards under data/.
Each row contains image: {bytes, path}, captions, url, physical
source_shard and source_row, and the scan metadata. Failed probes retain
null images and their failure status. Filter url_status == "ok" for training.
Rust stages successful downloads while Python writes bounded Arrow row groups;
images are fetched only once per successful batch.
Optional --recompress-images (SDK: recompress_images=True) losslessly
recompresses captured JPEGs to JPEG XL. Install the reference cjxl and djxl
tools on PATH; missing tools fail the scan before downloading a batch. Rust
invokes the codecs without a shell, limits each codec to one thread, reconstructs
the JPEG, and compares the reconstructed file byte for byte with the download.
Only smaller, verified results replace originals. Other formats, codec failures,
and conversions that do not save space retain the original bytes. This is JPEG
bitstream recompression, not lossy pixel re-encoding.
--recompress-workers (SDK: recompress_workers, default 2) separately bounds
encoding concurrency per scan invocation. Codec calls have a 60-second timeout
each. Rust performs staging, hashing, verification, and cleanup with the Python
GIL released. The codec implementation is supplied by the external libjxl tools.
Captured rows include stored_format (jxl or original), stored_bytes,
stored_sha256, and recompression_status; the existing download_* fields
always describe the original response. Failed downloads have null storage fields.
JXL bytes require a compatible training decoder or JPEG reconstruction with
djxl; webshart returns the stored bytes unchanged. Enabling recompression
requires a fresh scan destination so a resume cannot silently mix policies.
To upload each completed shard and discard its local image copy, add:
--capture-images --upload-repo owner/dataset --no-local-images
The destination repository must already exist. Uploads use huggingface_hub
authentication and endpoint configuration; --upload-endpoint overrides only
the upload endpoint. --upload-repo-type model supports repositories exposed
through the model API. The SDK equivalents are capture_images=True,
upload_repo="owner/dataset", keep_local_images=False, and optional
upload_endpoint, upload_token, and upload_repo_type arguments to
scan_url_metadata. Explicit upload credentials do not change source reads.
This mode needs temporary disk space for one download batch and its Parquet
shard per process (bounded by batch size times the maximum image size, plus
encoding overhead). It retains local metadata and checkpoints. Images are
removed only after a successful Hub commit and a durable local checkpoint.
A failed upload retains the completed batch for retry without refetching.
Resuming trusts recorded Hub commit receipts; keep the uploaded files intact.
The destination data/ files are self-contained image/caption Parquet shards;
the metadata overlay still refers to the original URL source. Local captured
shards can be loaded with discover_parquet_dataset("./recap-metadata/data").
For a remote captured repository, use subfolder="data" and optionally
endpoint and repo_type="model" in discover_parquet_dataset.
Use a fresh scan destination when enabling capture on an existing metadata-only
scan: discarded image bytes must be downloaded again. Use separate upload
repositories for different sources or revisions, and disjoint shard ranges
for concurrent writers to one repository.
For a separate uploader, give each producer a distinct child scan directory and
matching staging directory, then add --defer-uploads --no-local-images with
--upload-repo. For example, metadata in ./scans/000000 can stage bytes in
./buffer/000000 using --staging-directory. Downloads, native recompression,
and image Parquet writing continue without waiting for Hub uploads.
--min-staging-free-bytes pauses producers when the staging filesystem is low
on free space. Keep staging directories exclusive to their corresponding scans.
A separate process can periodically run:
webshart upload-capture-queue \
--destination ./scans --staging-directory ./buffer \
--upload-repo owner/dataset --min-bytes 1099511627776
The uploader consumes completed child scans once at least 1 TiB is pending.
It retains individual Parquet files and uploads groups of at most 64 files per
commit, with four transfer threads by default. Only one uploader may operate on
a queue. The SDK equivalent is upload_capture_queue; its byte threshold,
commit size, and transfer concurrency are configurable. Run with --flush after
all producers finish to upload the final remainder. If resuming an interrupted
batch, repeat until the returned pending_bytes is zero. Capture completion alone
is not proof of upload completion.
Active batch state and commit receipts remain with the scan metadata. A small receipt marker is committed alongside each remote file group, allowing retries to recognize a successful commit even if its acknowledgment was lost. Staged files are removed after the receipt and checkpoints are saved. Preserve both metadata and staging across restarts; missing pending image files cause an error rather than silently skipping rows. Deferred mode is part of scan identity; existing immediate-upload scans require an explicit checkpoint migration or a fresh destination before changing modes.
For independent workers claiming source shards dynamically, use the object-store shard queue. Conditional writes give each attempt a separate output prefix and prevent a replaced owner from publishing its result. Keep renewing claims until their outputs and manifest are uploaded; the shard queue coordinates ownership separately from the upload queue.
The resulting webshart-parquet.json and metadata/ files can be used locally
or hosted together in a metadata-only Hub repository. Keep scan-state.json,
checkpoints/, and parts/ locally for resuming. Each completed invocation
publishes an overlay covering the rows attempted so far, including partial
shards; unscanned rows retain their original metadata.
dataset = webshart.discover_parquet_dataset(
"UCSC-VLAA/Recap-DataComp-1B",
metadata="./recap-metadata",
)
loader = webshart.ParquetDataLoader(dataset, fetch_external_urls=True)
metadata = loader.get_metadata(0, 0)
if metadata.get("url_status") == "ok":
image = loader.load_sample(0, 0)
Overlay rows include width, height, url_status, url_http_status,
download_bytes, and download_sha256. scan-summary.json separates successful
geometry reads, HTTP status codes, timeouts, network errors, size limits and
invalid images. These are observations from one scan: access denial, temporary
failures, and unsupported formats are not proof of permanent link rot. A prefix
pilot is not a representative estimate of availability across the whole dataset.
Image URLs can change even when the Parquet source revision is pinned, and a
successful scan does not guarantee a later training request will succeed.
Metadata manifest format
Pass a local metadata directory or Hub dataset repository containing
webshart-parquet.json. The source files stay unchanged. For example:
{
"format": "webshart-parquet",
"version": 1,
"source": {"repo_id": "owner/source", "revision": "<40-character commit SHA>"},
"columns": {"image": "image", "captions": "instruction", "width": "width", "height": "height"},
"metadata_mode": "overlay",
"shards": [{
"path": "data/part-000.parquet",
"num_rows": 2,
"metadata_path": "custom/part-000.json"
}]
}
For URL sources, set "image_url": "url" in columns instead of "image".
This declares the payload column but does not enable network loading; the loader
still requires fetch_external_urls=True. Overlays cannot replace payload URLs.
The JSON metadata file is a list of row objects:
[
{"source_row": 1, "captions": [{"scene": "sunset"}, "warm lighting"], "rating": 5},
{"source_row": 0, "captions": {"scene": "forest", "lighting": "soft"}}
]
Alternatively, metadata_path can name a Parquet file with a source_row
integer column and arbitrary custom columns. Row order and source IDs do not
matter: the join uses the source shard path plus zero-based physical row.
Optional source_path values must match the manifest. Duplicates, invalid row
positions, unsafe paths, mismatched file sizes/row counts, and incorrect optional
metadata_sha256 checksums fail explicitly. size, when present, is the source
Parquet file size in bytes. Remote manifests must pin their source commit;
metadata_revision can additionally pin the metadata repository commit.
overlay merges custom values over source non-binary columns and permits sparse
overrides. Custom captions, width, and height override configured aliases.
replace reads metadata only from the companion file and requires every row to
be present. It avoids reading upstream metadata columns. Payload columns cannot
be overridden. Metadata is cached per shard with a bounded LRU (default: one
shard; adjust metadata_cache_size). Recreate the dataset after changing files.
Custom companion rows are indexed on temporary disk and read back by physical
row. Overlay indexes are retained up to metadata_cache_size; eviction closes
the least recently used index.
Parquet structs, lists, and maps retain their native Python representations.
For mixed strings/objects that Parquet cannot represent in one homogeneous
column, store JSON text and declare "json_columns": ["captions"] in the
manifest (or discovery options). Only explicitly declared columns are parsed;
ordinary caption strings are not guessed or stringified. Decimal values remain
decimal.Decimal; date/time values use their Parquet integer units.
Streaming and access costs
loader.iter_shard(shard_index, start=0, stop=None) streams selected image
columns page by page, reusing dictionaries. load_sample(shard_index, row_index)
skips preceding row groups but scans earlier rows within the selected group.
Unlike indexed tar, compressed Parquet does not provide constant-cost
per-image byte offsets. Prefer sequential iteration for large row groups.
Memory scales with selected columns' decoded pages/dictionaries, not an entire
shard; oversized dictionaries can still require hundreds of MiB. The
max_page_bytes discovery option limits individual compressed page/footer reads
(default 512 MiB), not total or decompressed memory.
Binary image columns and Hugging Face {bytes, path} image structs are supported.
Missing bytes produce sample.data is None; original structs remain in
sample.payloads. External URLs/path-only images are never fetched
automatically. Keep URL/hash metadata and apply your own retrieval policy.
The loader does not rewrite or download whole source shards, and does not expose
tar-only byte-offset, bucket, shard-cache, or field-rewrite APIs.
Why is indexed tar fast?
Problem: Standard tar files require sequential reading. To get file #10,000, you must read through files #1-9,999 first.
Solution: The indexed format stores byte offsets and sample metadata in a separate JSON file, enabling:
- HTTP range requests for any file
- True random access over network
- Parallel reads from multiple shards
- Large scale, aspect-bucketed datasets
- No wasted bandwidth
The Rust implementation provides:
- Real parallelism (no Python GIL)
- Zero-copy operations where possible
- Efficient HTTP connection pooling
- Optimized tokio async runtime
- Optional local caching for metadata and shards
- Fast aspect bucketing for image data
Datasets Using This Format
I discovered after creating this library that cheesechaser is the origin of the indexed tar format, which webshart has formalised and extended to include aspect bucketing support.
NebulaeWis/e621-2024-webp-4Mpixelpicollect/danbooru2(subfolder:images)webshart/OpenVid-1M-webshart-indices(indices forDev-Jahn/OpenVid-1M-wds)- Many picollect image datasets
- Your dataset could be next! See "Creating Indices" above
Requirements
- Python 3.12+
- Linux/macOS/Windows
Roadmap
- image decoding is currently not handled by this library, but it will be added with zero-copy.
- more informative API for caching and other Rust implementation details
- multi-gpu/multi-node friendly dataloader
Projects using webshart
- CaptionFlow uses this library to solve memory use and seek performance issues typical to webdatasets
License
MIT
Metadata
Release files for webshart 0.9.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| webshart-0.9.0.tar.gz | 257.3 kB | Without provenance | Details for webshart-0.9.0.tar.gz |
Built distributions (wheels)
| File | Actions Reset | |||
|---|---|---|---|---|
| webshart-0.9.0-cp312-abi3-win_amd64.whl | CPython 3.12 | abi3 | Windows x86-64 | Details for webshart-0.9.0-cp312-abi3-win_amd64.whl |
| webshart-0.9.0-cp312-abi3-manylinux_2_39_x86_64.whl | CPython 3.12 | abi3 | Linux glibc 2.39+ x86-64 | Details for webshart-0.9.0-cp312-abi3-manylinux_2_39_x86_64.whl |
| webshart-0.9.0-cp312-abi3-manylinux_2_39_aarch64.whl | CPython 3.12 | abi3 | Linux glibc 2.39+ ARM64 | Details for webshart-0.9.0-cp312-abi3-manylinux_2_39_aarch64.whl |
| webshart-0.9.0-cp312-abi3-manylinux_2_35_x86_64.whl | CPython 3.12 | abi3 | Linux glibc 2.35+ x86-64 | Details for webshart-0.9.0-cp312-abi3-manylinux_2_35_x86_64.whl |
| webshart-0.9.0-cp312-abi3-macosx_11_0_arm64.whl | CPython 3.12 | abi3 | macOS 11.0+ ARM64 | Details for webshart-0.9.0-cp312-abi3-macosx_11_0_arm64.whl |
Total release size: 37.0 MB
Release files / webshart-0.9.0.tar.gz
| Download URL | webshart-0.9.0.tar.gz |
|---|---|
| Size | 257.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
739927f0d02595f317f815fd3ea0123b3efd0e697d625df3a6bb7b6ae60c4ec3
|
|
BLAKE2b-256 checksum How to use checksums |
9f1de0cfb0d364ddbd863e4d6666a4160d68ca7d071c923495e2a2c503328472
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Release files / webshart-0.9.0-cp312-abi3-win_amd64.whl
| Download URL | webshart-0.9.0-cp312-abi3-win_amd64.whl |
|---|---|
| Size | 7.2 MB |
| Tags | CPython 3.12 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
6c6f9edc4b4ef7c06ac4f6426f31e1b8a21366ad77b6c85decb75581f000d3b6
|
|
BLAKE2b-256 checksum How to use checksums |
60c9c9a2922980e6815f2de3ad5a399864e43ad8dbc3593a43b72f14f9613a04
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Release files / webshart-0.9.0-cp312-abi3-manylinux_2_39_x86_64.whl
| Download URL | webshart-0.9.0-cp312-abi3-manylinux_2_39_x86_64.whl |
|---|---|
| Size | 7.6 MB |
| Tags | CPython 3.12 Linux glibc 2.39+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
a763f596862ee0113f53cfab42d83c8360b3c52901cd02640cb3bcffa65030bd
|
|
BLAKE2b-256 checksum How to use checksums |
7ba328dd0d58254eaafcfb3f550e4cf7660d8cc99be61725c46ca33b104533e7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Release files / webshart-0.9.0-cp312-abi3-manylinux_2_39_aarch64.whl
| Download URL | webshart-0.9.0-cp312-abi3-manylinux_2_39_aarch64.whl |
|---|---|
| Size | 7.2 MB |
| Tags | CPython 3.12 Linux glibc 2.39+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
026d3570821279f9174fdfef9531ccb0896a14a490634203f84199f729941ae6
|
|
BLAKE2b-256 checksum How to use checksums |
23cc17283e7cdc489d798f84250b0f0bd59532c022d0caf3dda254a6228f4f2c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Release files / webshart-0.9.0-cp312-abi3-manylinux_2_35_x86_64.whl
| Download URL | webshart-0.9.0-cp312-abi3-manylinux_2_35_x86_64.whl |
|---|---|
| Size | 7.6 MB |
| Tags | CPython 3.12 Linux glibc 2.35+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
29c2f8ddfea01ebd332becfb1ce1e16ea09b03f68c7ded5d97a0d6c253bced46
|
|
BLAKE2b-256 checksum How to use checksums |
e5ddecfece27c509ba10b5ea7cbc1800450de15881dbfdfb8905b5ca9f98fea0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Release files / webshart-0.9.0-cp312-abi3-macosx_11_0_arm64.whl
| Download URL | webshart-0.9.0-cp312-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 7.0 MB |
| Tags | CPython 3.12 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
05355f0a201ef1e4dc2d2bbc6e0f94716de45c74120a2c2394781f69c41c28d0
|
|
BLAKE2b-256 checksum How to use checksums |
adf843c58726142f952fa27735e06e9fe9e4024ef5894d6483b00e31f0d1c775
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.12.8
|
Security for webshart 0.9.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.