Skip to main content

Adapt generators and other iterables to a file-like interface

Project description

iterable-io

Status Version Python

iterable-io is a small Python library that provides an adapter so that it's possible to read from iterable objects in the same way as file-like objects.

It is primarily useful as "glue" between two incompatible interfaces. As an example, in the case where one interface expects a file-like object to call .read() on, and the other only provides a generator of bytes.

One way to solve this issue would be to write all the bytes in the generator to a temporary file, then provide that file instead, but if the generator produces a large amount of data then this is both slow to start, and resource-intensive.

This library allows streaming data between these two incompatible interfaces so as data is requested by .read(), it's pulled from the iterable. This keeps resource usage low and removes the startup delay.

Installation

pip install iterable-io

Documentation

The functionality of this library is accessed via a single function: open_iterable().

open_iterable() is designed to work the same was as the builtin open(), except that it takes an iterable to "open" instead of a file. For example, it can open the iterable in binary or text mode, has options for buffering, encoding, etc. See the docstring of open_iterable for more detailed documentation.

Simple examples

The following examples should be enough to understand in which cases open_iterable() would be useful and get a high-level understanding of how to use it:

Read bytes from a generator of bytes:

gen = generate_bytes()

# adapt the generator to a file-like object in binary mode
# (fp.read() will return bytes)
fp = open_iterable(gen, "rb")

while chunk := fp.read(4096):
    process_chunk(chunk)

Read lines of text from a generator of bytes:

gen = generate_bytes()

# adapt the generator to a file-like object in text mode
# (fp.read() will return a string, fp.readline is also available)
fp = open_iterable(gen, "rt", encoding="utf-8")

for line in fp:
    process_line_of_text(line)

Tests

This package contains extensive tests. To run them, install pytest (pip install pytest) and run py.test in the project directory.

License

Licensed under the GNU LGPLv3.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

iterable-io-1.0.0.tar.gz (6.0 kB view details)

Uploaded Source

Built Distribution

iterable_io-1.0.0-py3-none-any.whl (4.9 kB view details)

Uploaded Python 3

File details

Details for the file iterable-io-1.0.0.tar.gz.

File metadata

  • Download URL: iterable-io-1.0.0.tar.gz
  • Upload date:
  • Size: 6.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for iterable-io-1.0.0.tar.gz
Algorithm Hash digest
SHA256 fb9e1b739587a9ba0d5c60a3e1eb71246761583bc9f18b3c35bb112b44b18c3c
MD5 d6027af37f8e15edd346bdff6d5c5e3c
BLAKE2b-256 40be27d59b5c1d74ecbd26c1142f84b61d6cb04f0d0337697149645f34406b2d

See more details on using hashes here.

File details

Details for the file iterable_io-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: iterable_io-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 4.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.11.4

File hashes

Hashes for iterable_io-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 509b0e54888d7302d67687739d96014a79d54771723e9b0842a0243b467e6294
MD5 c87d4f59e271c2e1c6dcef24fa8fa04f
BLAKE2b-256 8d6339881aeeaf0a186fce8a755ec26e76ba61731a89b66e08b9139cec26ff0d

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page