GuideLoom
Flexible data. Reliable training. Built to scale.
GuideLoom is dataset storage and a resumable PyTorch DataLoader for large-scale training. Keep documents, token streams, and span annotations in one data pipeline, with global shuffling, independent distributed reads, and exact checkpointing.
Quick-start · Documentation · Design story
Why GuideLoom?
| Benefit | What you get |
|---|---|
| Support complex data layouts | Whole documents, fixed-length streams, or streams with variable-size span annotations. Keep concept lists and document IDs connected to the training windows they describe. |
| Your schema, your context window | Custom encoders and decoders for documents and metadata; multiple aligned stream fields for tokens, loss masks, or per-token concepts. Select stream window sizes at read time. |
| Instant global shuffling | Seeded sampling without replacement across all shards, without allocating a dataset-sized permutation or storing a shuffled dataset copy for each epoch. |
| Exact resumption with prefetching | Checkpoint the position delivered to training while workers prepare future batches. Resume at the next batch with the same dataset, seed, and batch partitioning. |
| Independent distributed reads | Each rank reads its own samples from local or shared storage. No communication between loaders or whole-shard staging on every node. Designed for HPC parallel filesystems, with extensible read backends. |
| High performance from the ground up | NumPy buffer views for compatible encodings, GIL-free compiled indexing and shuffling, plus configurable workers and background prefetching to overlap data preparation with training. |
See performance tips for encoding and I/O tuning, and save and resume for checkpointing examples and the settings needed to preserve sample order.
Requires Python 3.13 or 3.14. See the compatibility policy.
Install
uv venv --python 3.13
uv pip install "guideloom[torch]"
The torch extra installs PyTorch for StatefulDataLoader.
Dataset storage and readers are available from the base package:
uv pip install guideloom.
For the unreleased version, install from a checkout with uv pip install -e ".[torch]".
Learn GuideLoom
Start with the quick-start: write a dataset once in a preprocessing script, then open it later in a training script. The documentation follows that workflow:
- Beginner: choose a storage format, write independent shards in parallel, and load PyTorch batches from storage across ranks.
- Intermediate: save and resume, learn NumPy structured arrays, and define encodings for each storage format.
- Advanced: tune performance, use dataset and async readers directly, implement I/O backends, and integrate another framework such as JAX.
View documentation locally
From a repository checkout, use uv, Python 3.13, and Node.js 24. Run these setup commands from the repository root:
uv venv --python 3.13
uv pip install --group docs
uv run --no-sync npm --prefix docs/components ci
Start the local preview server:
uv run --no-sync npm --prefix docs/components run serve
Open http://localhost:8000/ in your browser. Keep the
server running while you edit files under docs/; it rebuilds the site and
reloads the browser automatically, including changes to the interactive components.
Press Ctrl+C in the terminal to stop it.
The documentation tools do not require GuideLoom's runtime dependencies.
License
GuideLoom is licensed under the Apache License, Version 2.0. See NOTICE and licenses/ for third-party attributions and license terms.
Metadata
Release files for guideloom 1.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| guideloom-1.1.0.tar.gz | 113.9 kB | With provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| guideloom-1.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 178.0 kB
Release files / guideloom-1.1.0.tar.gz
| Download URL | guideloom-1.1.0.tar.gz |
|---|---|
| Size | 113.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9416809b0634cbd0f2a88fd2f09ae8e113333065fc4a3e81d7bbd28df24a3494
|
|
BLAKE2b-256 checksum How to use checksums |
26f1032ebdef6bfb53cfb3759501cd40d075672b239bb09b6ae496e308541358
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 24, 2026.
Transparency logRelease files / guideloom-1.1.0-py3-none-any.whl
| Download URL | guideloom-1.1.0-py3-none-any.whl |
|---|---|
| Size | 64.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c8e03bac68fe79e5b9d2cbf3e16d3f98c5b11767c7fd9a450c9988fd033ae64c
|
|
BLAKE2b-256 checksum How to use checksums |
57dc5897416823a710d8b1552bed339dbdc784e8d1a4210de064aae0e7154a6a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 24, 2026.
Transparency logSecurity for guideloom 1.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for guideloom-1.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 24, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
