Adapt generators and other iterables to a file-like interface
Project description
iterable-io
iterable-io
is a small Python library that provides an adapter so that it's possible to read from
iterable objects in the same way as
file-like objects.
It is primarily useful as "glue" between two incompatible interfaces. As an example, in the case
where one interface expects a file-like object to call .read()
on, and the other only provides a
generator of bytes.
One way to solve this issue would be to write all the bytes in the generator to a temporary file, then provide that file instead, but if the generator produces a large amount of data then this is both slow to start, and resource-intensive.
This library allows streaming data between these two incompatible interfaces so as data is requested
by .read()
, it's pulled from the iterable. This keeps resource usage low and removes the startup
delay.
Installation
pip install iterable-io
Documentation
The functionality of this library is accessed via a single function: open_iterable()
.
open_iterable()
is designed to work the same was as the builtin open()
, except that it takes an
iterable to "open" instead of a file. For example, it can open the iterable in binary or text mode,
has options for buffering, encoding, etc. See the docstring of open_iterable
for more detailed
documentation.
Simple examples
The following examples should be enough to understand in which cases open_iterable()
would be
useful and get a high-level understanding of how to use it:
Read bytes from a generator of bytes:
gen = generate_bytes()
# adapt the generator to a file-like object in binary mode
# (fp.read() will return bytes)
fp = open_iterable(gen, "rb")
while chunk := fp.read(4096):
process_chunk(chunk)
Read lines of text from a generator of bytes:
gen = generate_bytes()
# adapt the generator to a file-like object in text mode
# (fp.read() will return a string, fp.readline is also available)
fp = open_iterable(gen, "rt", encoding="utf-8")
for line in fp:
process_line_of_text(line)
Tests
This package contains extensive tests. To run them, install pytest
(pip install pytest
) and run
py.test
in the project directory.
License
Licensed under the GNU LGPLv3.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file iterable-io-1.0.0.tar.gz
.
File metadata
- Download URL: iterable-io-1.0.0.tar.gz
- Upload date:
- Size: 6.0 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb9e1b739587a9ba0d5c60a3e1eb71246761583bc9f18b3c35bb112b44b18c3c |
|
MD5 | d6027af37f8e15edd346bdff6d5c5e3c |
|
BLAKE2b-256 | 40be27d59b5c1d74ecbd26c1142f84b61d6cb04f0d0337697149645f34406b2d |
File details
Details for the file iterable_io-1.0.0-py3-none-any.whl
.
File metadata
- Download URL: iterable_io-1.0.0-py3-none-any.whl
- Upload date:
- Size: 4.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.11.4
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 509b0e54888d7302d67687739d96014a79d54771723e9b0842a0243b467e6294 |
|
MD5 | c87d4f59e271c2e1c6dcef24fa8fa04f |
|
BLAKE2b-256 | 8d6339881aeeaf0a186fce8a755ec26e76ba61731a89b66e08b9139cec26ff0d |