Random Access Archive
.raa files are essentially a dict header + consecutive bytes of the samples. It was made to faccilitate and accelerate deep learning training on large datasets. It's written in Rust and fast, but easily accesible programmatically in Python. Most importantly, it allows you to shuffle the data, without sacrificing too much on sequential reads, by shuffling blocks of contiguous data. It also allows for lazy sharding.
Comparison
The main advantage of this library, is how extensible it is. Other libraries like Webdataset, FFCV, Streaming Dataset, TF Record, are very batteries included, which is great for experimentation, but sacrifices on extensibility heavily since they also include data processing. Our philosiphy quite simple, you write string byte pairs, and you read string byte pairs. We only implement functionality that NEEDS to be implemented at the reader level for optimization, like shuffling and sharding.
Benchmarks:
!todo
Usage
pip install rand-archive
Writing:
from rand_archive import Writer
with Writer("test.raa") as w:
w.write("test", bytes("test"))
Reading
from rand_archive import Reader
for _ in Reader().open_file("dummy.raa").with_shuffling():
pass
1 maintainer
Release files for rand-archive 0.2.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| rand_archive-0.2.3.tar.gz | 24.3 kB | Details |
Built distributions (wheels)
Total release size: 115.6 MB
Release files / rand_archive-0.2.3.tar.gz
| Download URL | rand_archive-0.2.3.tar.gz |
|---|---|
| Size | 24.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a2bdcb15757a898385a61585fbd455c31e17839381bb86b829d2c67dd8d87a30
|
|
BLAKE2b-256 checksum How to use checksums |
ea6afe8c84396274585b1aa5288f95fcb01fed76cfce02d4fc267362535fa17c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rand_archive-0.2.3-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 5.0 MB |
| Tags | Linux glibc 2.17+ x86-64 PyPy 3.10 PyPy 3.10 7.3 |
|
SHA-256 checksum How to use checksums |
fbae0d8fe4470d687381e7ff8697da9894873617a9eb56324b9a04a2c3439365
|
|
BLAKE2b-256 checksum How to use checksums |
b83749e39f8a18da888368dcb4385938e7cc1a45fb45c8d2c776474b4e8834aa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | rand_archive-0.2.3-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.5 MB |
| Tags | Linux glibc 2.17+ ARM64 PyPy 3.10 PyPy 3.10 7.3 |
|
SHA-256 checksum How to use checksums |
29dda304dac62672200be0cd70a86b4177cdaf0dc1be206f3a34e0f52d10c42f
|
|
BLAKE2b-256 checksum How to use checksums |
845968052a93a63662628c0ca966836a8df6942d1d7046371df12ba3c74940a4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rand_archive-0.2.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 5.0 MB |
| Tags | Linux glibc 2.17+ x86-64 PyPy 3.9 PyPy 3.9 7.3 |
|
SHA-256 checksum How to use checksums |
5e13185362a070f90e738e6b5a38350467a04162aeaa87750bbf3e647479ce55
|
|
BLAKE2b-256 checksum How to use checksums |
bd208de177257a65dc8e97c709cfcdcc9d69cbd554cabe3fd5c246d7bc402092
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | rand_archive-0.2.3-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.5 MB |
| Tags | Linux glibc 2.17+ ARM64 PyPy 3.9 PyPy 3.9 7.3 |
|
SHA-256 checksum How to use checksums |
88dd1153697e526c82a3cc33d145675289c6f1b33d0b1bfe2171a71937ce7ecc
|
|
BLAKE2b-256 checksum How to use checksums |
bdb5e6c54cf995bc67e7c62f29c9f856a03ece2dd186a20c9c44793df12106aa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rand_archive-0.2.3-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 5.0 MB |
| Tags | Linux glibc 2.17+ x86-64 PyPy 3.8 PyPy 3.8 7.3 |
|
SHA-256 checksum How to use checksums |
bbd76e42083082ce3a86da01060ec01e7dd0f768c5fd75c97d42c897e15499e7
|
|
BLAKE2b-256 checksum How to use checksums |
3b413dc447524b1c14b18654b6e7d69ae9697c792201d687248c03ee6b7736cd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | rand_archive-0.2.3-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.5 MB |
| Tags | Linux glibc 2.17+ ARM64 PyPy 3.8 PyPy 3.8 7.3 |
|
SHA-256 checksum How to use checksums |
468f689bbe620b87f9e912340400a3bc4ab050a13051528988f7343360601a46
|
|
BLAKE2b-256 checksum How to use checksums |
87e5695825a9671a39cbc3ab5d927809814837a224d3398ac8a1444e13d38f9c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rand_archive-0.2.3-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 5.0 MB |
| Tags | Linux glibc 2.17+ x86-64 PyPy 3.7 PyPy 3.7 7.3 |
|
SHA-256 checksum How to use checksums |
5a41c86a5ad2fb18cc642cb27095d36741863fdcd0678b1daa3149a5e7c5f622
|
|
BLAKE2b-256 checksum How to use checksums |
583bd237e1617caa3c801a0e47cebdfadc7752f7d913d99461af644c18258786
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | rand_archive-0.2.3-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.5 MB |
| Tags | Linux glibc 2.17+ ARM64 PyPy 3.7 PyPy 3.7 7.3 |
|
SHA-256 checksum How to use checksums |
1c90a80a57f8774810e3c41dc595bfb8f60b9a26a046e87dc439791578762f15
|
|
BLAKE2b-256 checksum How to use checksums |
ccf916322e21e092a5cdc4ca171eec017138cc7a1943211be92ecec242108cf4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rand_archive-0.2.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 5.0 MB |
| Tags | CPython 3.12 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
63c715311e6c533c1e960f1651cc3dc534e9d69c334172eaae455a20eada2252
|
|
BLAKE2b-256 checksum How to use checksums |
4868cfe351038e784ad7ef2452858cbee36707097b078f3aea673003f153f222
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | rand_archive-0.2.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.5 MB |
| Tags | CPython 3.12 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
a3c7e7f9306bc2b740bc46288475f2a2bd91f0960807ff9a661dac03f2a57874
|
|
BLAKE2b-256 checksum How to use checksums |
201448afec1d13a4b1820811877221d2de2ae7beacc0c9808e2b2affefb255b8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp311-none-win_amd64.whl
| Download URL | rand_archive-0.2.3-cp311-none-win_amd64.whl |
|---|---|
| Size | 3.0 MB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
71ee87ac4ebd4ad87f5756ed00112c46b2780101a288d23b341b50ccf36e57b0
|
|
BLAKE2b-256 checksum How to use checksums |
957a51f28006bdec5294402ee851ec4a70f160ed5a15dbff4cf3385a904e3e20
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rand_archive-0.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 5.0 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
61704854de332b5f40ba6c18399d0f99f3d61304cd2773445518391f7e686a8f
|
|
BLAKE2b-256 checksum How to use checksums |
07d49c32c525204ecd311cc4d50405afe37a257d547a8556c17856689c24076d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | rand_archive-0.2.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.5 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
743fff5cbaa5c05c31dc489cf6dcfe89febddb8f663c51fc7066e92af4a99c97
|
|
BLAKE2b-256 checksum How to use checksums |
f6a49ceaa40572d15435fc9c8e5e6e7a5d286a0c9f6aee4831c4389c76f02423
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp311-cp311-macosx_11_0_arm64.whl
| Download URL | rand_archive-0.2.3-cp311-cp311-macosx_11_0_arm64.whl |
|---|---|
| Size | 3.0 MB |
| Tags | CPython 3.11 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
3c066d7b8e9a6418e8786ff306639d031e0ca347ba0d0e699cf45256b80e90ac
|
|
BLAKE2b-256 checksum How to use checksums |
adfd5d89024693cdc5979262b3ecc85a09dbccbe58b527804b3768fc3f46f30d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp310-none-win_amd64.whl
| Download URL | rand_archive-0.2.3-cp310-none-win_amd64.whl |
|---|---|
| Size | 3.0 MB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
92d79f908ecedce1fdb075a7eba41ae234ddfa99da8f41aef2e7d721a126226c
|
|
BLAKE2b-256 checksum How to use checksums |
47099e966cdf934630272456715fd2378f3a9aaa5efc01a71b0458d608aa4045
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rand_archive-0.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 5.0 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
3964f6061f4ff6479214f66b3dfbfa2029fb6834990bc1d46a03f1be48032976
|
|
BLAKE2b-256 checksum How to use checksums |
76a42f2c91cf7a0604078bede1d916f48fa9e7685f18da58b8cd3ec82e031768
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | rand_archive-0.2.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.5 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
02611b80c870fef2428642d83b181397472c69a8a072b1fa1596ea52c9c7b5ba
|
|
BLAKE2b-256 checksum How to use checksums |
aab2ea4de3a533411133416fc4142bb2d07e158dc47e6bf13de2b104c747bf74
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp310-cp310-macosx_11_0_arm64.whl
| Download URL | rand_archive-0.2.3-cp310-cp310-macosx_11_0_arm64.whl |
|---|---|
| Size | 3.0 MB |
| Tags | CPython 3.10 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
5aa88e301b7f15d5ab5323400f953c63ec77c0a378e406f0a209a8a54aef1ea5
|
|
BLAKE2b-256 checksum How to use checksums |
16b096cc9fc5361308cef7adff0963c4caed15ad6b2cec2234702e9b426b2416
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp39-none-win_amd64.whl
| Download URL | rand_archive-0.2.3-cp39-none-win_amd64.whl |
|---|---|
| Size | 3.0 MB |
| Tags | CPython 3.9 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
2a3ba2a32514cd10d3454d2a804672ddb3b49e4b0d05e6229b9803e7b2b4d8e9
|
|
BLAKE2b-256 checksum How to use checksums |
c0c552a4a89c303d60497f4afb05799a6053df6d85c4536110cafd25643b9275
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rand_archive-0.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 5.0 MB |
| Tags | CPython 3.9 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
c52418b9278b511b37af72d0888a2e4ab05fc67c55ea1505771b0cf7d41c1fe5
|
|
BLAKE2b-256 checksum How to use checksums |
24d5a1e410018343e3381c7b51283c2e4f54a21cf9ab9cebb7153b3ac56fb323
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | rand_archive-0.2.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.5 MB |
| Tags | CPython 3.9 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
ac3c9aa1f9b48411562e4ebe3e99c2d48b878cb44545606779606971270fd1e5
|
|
BLAKE2b-256 checksum How to use checksums |
b6349a1a6076ada2a472af86d451949a9f46b9cb61d6b2c6f45af44f57141fa2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp38-none-win_amd64.whl
| Download URL | rand_archive-0.2.3-cp38-none-win_amd64.whl |
|---|---|
| Size | 3.0 MB |
| Tags | CPython 3.8 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
7dddd2dcad3c48ef7dc5a9458a8c9440dfa86eeeb00f1b6ea05d92b076cd3dc6
|
|
BLAKE2b-256 checksum How to use checksums |
d8b8e3dc973841472524c669f95a9ffb40a12ff03712ef201356c536907216bf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rand_archive-0.2.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 5.0 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
0be007156987f6ecd60f40ba4e00e75cf92169b049a7db7668b6f5015bcd8b38
|
|
BLAKE2b-256 checksum How to use checksums |
76f150dc8a5adda7810ae3e857a6ed4b686d5103116a50028ff095816b18f18c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | rand_archive-0.2.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.5 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
a7b9897e5a4365ea1387f7a76df504d45755c5d8595ce0e238fffa487ed395db
|
|
BLAKE2b-256 checksum How to use checksums |
6f0697e1e24b9aafe1caeadf83b2234d6ba44a10517020331b4ccb3e57c11b95
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp37-none-win_amd64.whl
| Download URL | rand_archive-0.2.3-cp37-none-win_amd64.whl |
|---|---|
| Size | 3.0 MB |
| Tags | CPython 3.7 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
f36ce3295e3bcc3b5255d7041cf9b6653264c592c903992fe7b5a3ffc8bf76b6
|
|
BLAKE2b-256 checksum How to use checksums |
70488e1585d4b7e943c478bc656bdddc9e991036eed5a5e8597e56b110d0e3d7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | rand_archive-0.2.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 5.0 MB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
926f5a01e0342acb8739d6ecc8083e1d1440b2b532ccef85291aa2a6bdae70d6
|
|
BLAKE2b-256 checksum How to use checksums |
7e87eb47249084450bf7f4aad3a5b31983c7eaff88d9322938400702d12caa3c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
Release files / rand_archive-0.2.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | rand_archive-0.2.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.5 MB |
| Tags | CPython 3.7 CPython 3.7 pymalloc Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
4b4350af3c6f7f331876e7bb38174526be0837423ecadf19000b5d11df13f817
|
|
BLAKE2b-256 checksum How to use checksums |
0b4d5907bff01b162272bc7059b0c2c4055dd92e0366bb5ad08a774ca0d6c12b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.2.3
|
