Fast Reading
A fast reading file library for Python
Description
Fast Reading is a high-performance file reading library for Python, written in Rust using pyo3. It provides an efficient mechanism for reading files by loading them in batches and yielding file content one at a time. This makes it ideal for processing large volumes of data with minimal overhead.
Features
- High Performance: Built with Rust for speedy file operations.
- Seamless Python Integration: Uses pyo3 for a natural Python interface.
- Batch File Reading: Reads files in batches to optimize I/O, then iterates over individual file data.
- Flexible Iterators: Provides different iterator types for various use cases.
- Cross-Platform Compatibility: Works with Python 3.9 and above.
Installation
Ensure you have maturin installed:
pip install fast-reading
Tests
> 100_000 files of 1024 bytes each
$ Python reading time: 0.777500 seconds
$ FilesBatchIterator reading time (batch_size=5): 0.449203 seconds
$ FlattenFilesBatchIterator reading time (batch_size=5): 0.448566 seconds
> 100_000 files of 32768 bytes each
$ Python reading time: 1.138894 seconds
$ FilesBatchIterator reading time (batch_size=5): 0.847089 seconds
$ FlattenFilesBatchIterator reading time (batch_size=5): 0.835050 seconds
> 1_000_000 files of 4096 bytes each
$ Python reading time: 14.950660 seconds
$ FilesBatchIterator reading time (batch_size=5): 5.086445 seconds
$ FlattenFilesBatchIterator reading time (batch_size=5): 5.068229 seconds
Usage
FilesBatchIterator Example
This iterator reads files in batches and returns a list of file contents for each batch.
from fast_reading import FilesBatchIterator
# Initialize the iterator for reading files from a directory with a batch size of 100
for file_bytes in FilesBatchIterator("/path/to/directory", batch_size=100):
# file_bytes is a list of bytes read from one or more files
print(file_bytes)
FlattenFilesBatchIterator Example
This iterator loads a batch of files but yields file content one by one.
from fast_reading import FlattenFilesBatchIterator
# Initialize the iterator to read files in batches of 5,
# but return one file's content at a time.
for file_content in FlattenFilesBatchIterator("/path/to/directory", batch_size=5):
# file_content is the bytes content of a single file
print(file_content)
License
This project is licensed under the MIT License.
Contact
Author: @vffuunnyy
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
fast_reading-0.1.2.tar.gz
(6.5 kB
view details)
Built Distributions
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file fast_reading-0.1.2.tar.gz.
File metadata
- Download URL: fast_reading-0.1.2.tar.gz
- Upload date:
- Size: 6.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e24531a47be8cecf931d6dde8e0dd6f3fbb06b4c326f34459643ac6a4428a7a1
|
|
| MD5 |
bdc7988fb45127dd86d17011a437931b
|
|
| BLAKE2b-256 |
4fc35823b055590b79b5b275e4c218c6f85500544bb5e708ee101265426fc64f
|
File details
Details for the file fast_reading-0.1.2-cp313-cp313-win_amd64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp313-cp313-win_amd64.whl
- Upload date:
- Size: 141.8 kB
- Tags: CPython 3.13, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2b2f314dc7cc1b20e115a0f8f3c6129eb554ff9418828c8593490505ca29d713
|
|
| MD5 |
e91ed404765e7609474f372b786be498
|
|
| BLAKE2b-256 |
8253e19976254b9f60c8cc622f7b414144e4f8ed12770f2e39a154bb884ed884
|
File details
Details for the file fast_reading-0.1.2-cp313-cp313-manylinux_2_34_x86_64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp313-cp313-manylinux_2_34_x86_64.whl
- Upload date:
- Size: 273.2 kB
- Tags: CPython 3.13, manylinux: glibc 2.34+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
0bba1c2c2fb691365b27484733402d6d782a1cfb4be2af34d4eb7102f7b87bce
|
|
| MD5 |
9bcc3c27839258bcd95480784306106e
|
|
| BLAKE2b-256 |
b436971a28abb4b4d2ec1fc72d0776532b870a0d01fda821cae80f7c1174e304
|
File details
Details for the file fast_reading-0.1.2-cp313-cp313-macosx_11_0_arm64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp313-cp313-macosx_11_0_arm64.whl
- Upload date:
- Size: 237.4 kB
- Tags: CPython 3.13, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
1f2fd018b74448b8670db808a40cded69f2443156ca5959ad0b1b8c18882cbdd
|
|
| MD5 |
5a576a76a318d3ef3f3676ae2b59ef65
|
|
| BLAKE2b-256 |
a0751590956b00c28fc894ec5630f90ac63c199fb79087a8d1c1cb03240c960d
|
File details
Details for the file fast_reading-0.1.2-cp312-cp312-win_amd64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp312-cp312-win_amd64.whl
- Upload date:
- Size: 142.1 kB
- Tags: CPython 3.12, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8ac8aef55870700d1182d88c269f4d34f110774e23b664300e732879194fbe72
|
|
| MD5 |
2389406a7048da671b22f2cacbec9ea9
|
|
| BLAKE2b-256 |
0be51e4353d9175bbfd97f66da88051861cbca22f0fbdc9a67c58e81c89a3ed1
|
File details
Details for the file fast_reading-0.1.2-cp312-cp312-manylinux_2_34_x86_64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp312-cp312-manylinux_2_34_x86_64.whl
- Upload date:
- Size: 273.3 kB
- Tags: CPython 3.12, manylinux: glibc 2.34+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e335fc3b1e62a5f5c720c7787fc202b4138d04a5b12bf3554beb7733200cd072
|
|
| MD5 |
d2856fa44f4ea960c1530252a3af4ec7
|
|
| BLAKE2b-256 |
6045a5bcb20da04e67edda1ec47e7a5c991eb1142e93794a316e6705d374ef4f
|
File details
Details for the file fast_reading-0.1.2-cp312-cp312-macosx_11_0_arm64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp312-cp312-macosx_11_0_arm64.whl
- Upload date:
- Size: 236.9 kB
- Tags: CPython 3.12, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
50554854fbda4b1156d17e2a824fed9b3d530d4b6f3a7a6e3eef8acbe8e060f9
|
|
| MD5 |
ae4fd8d97acb4b84bd3ebd4a9729dbc9
|
|
| BLAKE2b-256 |
22f091464c4535a50e2b38d7a4563afa1418d71f6fd898238c608c78a080633f
|
File details
Details for the file fast_reading-0.1.2-cp311-cp311-win_amd64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp311-cp311-win_amd64.whl
- Upload date:
- Size: 142.0 kB
- Tags: CPython 3.11, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b1a6ebf69b102acd2488a0475234211ea02864638b8f0ca4995d1be47645d1e1
|
|
| MD5 |
240aad6e39a952178ef309c9c3fc025b
|
|
| BLAKE2b-256 |
8244efb0b253cb012def63e35d58861be8296e77d6807889da3996d9b0b58bdd
|
File details
Details for the file fast_reading-0.1.2-cp311-cp311-manylinux_2_34_x86_64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp311-cp311-manylinux_2_34_x86_64.whl
- Upload date:
- Size: 273.5 kB
- Tags: CPython 3.11, manylinux: glibc 2.34+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2b9e75470297e5e3194cf0dc1de8fcf95f1ca9fa584f8f93871045c09567fe05
|
|
| MD5 |
1d004cf0049b909022df23d53dfc2a37
|
|
| BLAKE2b-256 |
ac5476fd24bde2ca22fce9d1d96da873392261da8d28b300eabbab997f540129
|
File details
Details for the file fast_reading-0.1.2-cp311-cp311-macosx_11_0_arm64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp311-cp311-macosx_11_0_arm64.whl
- Upload date:
- Size: 240.9 kB
- Tags: CPython 3.11, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8ffaf7fa70cd241b04c9447fcbae64fb0b1d78b568c1876a852d643e8781b515
|
|
| MD5 |
49c28a3f965ec38abc58cdb5540b84de
|
|
| BLAKE2b-256 |
cddb520063b0617cd1fb63d0f7da0515f47373b55f5f077a73045c8f74532462
|
File details
Details for the file fast_reading-0.1.2-cp310-cp310-win_amd64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp310-cp310-win_amd64.whl
- Upload date:
- Size: 142.0 kB
- Tags: CPython 3.10, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2f31a50f84936d6700586797afc60c4e7e332b63033750ec5aa9ce2c1294e2a6
|
|
| MD5 |
14357b3b07ebe9240860233fc173cde5
|
|
| BLAKE2b-256 |
4c4312e42aede84174ae03c30daa322f33fab95515a654990e806bf09f59ab79
|
File details
Details for the file fast_reading-0.1.2-cp310-cp310-manylinux_2_34_x86_64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp310-cp310-manylinux_2_34_x86_64.whl
- Upload date:
- Size: 274.0 kB
- Tags: CPython 3.10, manylinux: glibc 2.34+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8ece7f2031777037eed6919647aa1cfc77416a7be03636730bf83b640e52c12f
|
|
| MD5 |
c0f077ada129ae7907c7fc47519ff317
|
|
| BLAKE2b-256 |
ef5ad08b05a5dbe464365eac19ff53029e6b6c6e2ddfeb9361cf7f529641ee28
|
File details
Details for the file fast_reading-0.1.2-cp310-cp310-macosx_11_0_arm64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp310-cp310-macosx_11_0_arm64.whl
- Upload date:
- Size: 240.9 kB
- Tags: CPython 3.10, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
83c539b40c54cabc0b9538099564d53de7dd481db8aa0cb4677168e7036762a9
|
|
| MD5 |
0cb6ef1189122194f23b211957cfcfa3
|
|
| BLAKE2b-256 |
7fe52e03c7c0e5cddedabb7cf010924d592941e58d8347f7d9e06a2ead6cdf7f
|
File details
Details for the file fast_reading-0.1.2-cp39-cp39-win_amd64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp39-cp39-win_amd64.whl
- Upload date:
- Size: 142.0 kB
- Tags: CPython 3.9, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
782e3a6c97195f731155746767d5fa724f35562e95045adc4ad8604b0fb16d7d
|
|
| MD5 |
44840e8a155de3c91d655d80494ce1f4
|
|
| BLAKE2b-256 |
831d14c83bd7a036cd1b87c507f6d93740cb5d3e0becc4bc2c62320f5c42dbee
|
File details
Details for the file fast_reading-0.1.2-cp39-cp39-manylinux_2_34_x86_64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp39-cp39-manylinux_2_34_x86_64.whl
- Upload date:
- Size: 274.3 kB
- Tags: CPython 3.9, manylinux: glibc 2.34+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
123424ae6f6ae0e7a387cda1d6a227ecb932ca90a957ba7bbd130bd2207c1288
|
|
| MD5 |
ed789e4e53d54d2255ebc8481c3cf2e8
|
|
| BLAKE2b-256 |
5f9a990dcefe8f81f22d6e7a5894e2f228c9408e0104b3d9f17289d6deaf747c
|
File details
Details for the file fast_reading-0.1.2-cp39-cp39-macosx_11_0_arm64.whl.
File metadata
- Download URL: fast_reading-0.1.2-cp39-cp39-macosx_11_0_arm64.whl
- Upload date:
- Size: 241.4 kB
- Tags: CPython 3.9, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.12.9
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
aeaafc14d4b469694656db5e9f242bcd6d0c08b272a9d2900d8171ff2902fb59
|
|
| MD5 |
091a676fdcb50622e1d986cf584c242f
|
|
| BLAKE2b-256 |
0faa6e62bbc0f7e807ebd9820cfb7f7a63916633198de4f673cd47e763c1c7bd
|
