Adapt generators and other iterables to a file-like interface
Project description
iterable-io
iterable-io
is a small Python library that provides an adapter so that it's possible to read from
iterable objects in the same way as
file-like objects.
It is primarily useful as "glue" between two incompatible interfaces. As an example, in the case
where one interface expects a file-like object to call .read()
on, and the other only provides a
generator of bytes.
One way to solve this issue would be to write all the bytes in the generator to a temporary file, then provide that file instead, but if the generator produces a large amount of data then this is both slow to start, and resource-intensive.
This library allows streaming data between these two incompatible interfaces so as data is requested
by .read()
, it's pulled from the iterable. This keeps resource usage low and removes the startup
delay.
Installation
pip install iterable-io
Documentation
The functionality of this library is accessed via a single function: open_iterable()
.
open_iterable()
is designed to work the same was as the builtin open()
, except that it takes an
iterable to "open" instead of a file. For example, it can open the iterable in binary or text mode,
has options for buffering, encoding, etc. See the docstring of open_iterable
for more detailed
documentation.
Simple examples
The following examples should be enough to understand in which cases open_iterable()
would be
useful and get a high-level understanding of how to use it:
Read bytes from a generator of bytes:
gen = generate_bytes()
# adapt the generator to a file-like object in binary mode
# (fp.read() will return bytes)
fp = open_iterable(gen, "rb")
while chunk := fp.read(4096):
process_chunk(chunk)
Read lines of text from a generator of bytes:
gen = generate_bytes()
# adapt the generator to a file-like object in text mode
# (fp.read() will return a string, fp.readline is also available)
fp = open_iterable(gen, "rt", encoding="utf-8")
for line in fp:
process_line_of_text(line)
Tests
This package contains extensive tests. To run them, install pytest
(pip install pytest
) and run
py.test
in the project directory.
License
Licensed under the GNU LGPLv3.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for iterable_io-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 509b0e54888d7302d67687739d96014a79d54771723e9b0842a0243b467e6294 |
|
MD5 | c87d4f59e271c2e1c6dcef24fa8fa04f |
|
BLAKE2b-256 | 8d6339881aeeaf0a186fce8a755ec26e76ba61731a89b66e08b9139cec26ff0d |