to-file-like-obj
Python utility function to convert an iterable of bytes or str to a readable file-like object.
It can be seen as the inverse of the two-argument iter function. The iter function allows conversion of file-like objects to iterables, but the function here converts from iterables to file-like objects. This allows you to bridge the gap between incompatible streaming APIs - passing data from sources that offer data as iterables to destinations that only accept file-like objects.
Features
- Inherits from
IOBase- some APIs require this - The resulting file-like object is well-behaved - it does not return more data than requested
- It evaluates the iterable lazily - avoiding loading all its data into memory
- Under the hood copying is avoided as much as possible
- Converts iterables of
bytesto bytes-based file-like objects, which can be passed to boto3's upload_fileobj or to io.TextIOWrapper which is useful in stream CSV parsing. - Converts iterables of
strto text-based file-like objects, which can be passed to psycopg2's copy_expert
Installation
pip install to-file-like-obj
Usage
If you have an iterable of bytes instances, you can pass them to the to_file_like_obj function, and it will return the corresponding file-like object.
from to_file_like_obj import to_file_like_obj
f = to_file_like_obj((b'one', b'two', b'three',))
print(f.read(5)) # b'onetw'
print(f.read(6)) # b'othree'
If you have an iterable of str instances, you can pass them to the to_file_like_obj, along with base=str as a named argument, and it will return the corresponding file-like object.
from to_file_like_obj import to_file_like_obj
f = to_file_like_obj(('one', 'two', 'three',), base=str)
print(f.read(5)) # 'onetw'
print(f.read(6)) # 'othree'
These examples have the iterables hard coded and so loaded all into memory. However, to_file_like_obj works equally well with iterables that are generated dynamically, and without loading them all into memory.
Recipe: parsing a CSV file while downloading it
Using httpx it's possible to use the to_file_like_obj function to parse a CSV file while downloading it.
import csv
import io
import httpx
from to_file_like_obj import to_file_like_obj
with httpx.stream("GET", "https://www.example.com/my.csv") as r:
bytes_iter = r.iter_bytes()
f = to_file_like_obj(bytes_iter)
lines_iter = io.TextIOWrapper(f, newline='', encoding='utf=8')
rows_iter = csv.reader(lines):
for row in rows_iter:
print(row)
Recipe: parsing a zipped CSV file while downloading it
Similarly, using httpx and stream-unzip, it's possible to use the to_file_like_obj function to robustly parse a zipped CSV file while downloading it.
import csv
import io
import httpx
from stream_unzip import stream_unzip
from to_file_like_obj import to_file_like_obj
with httpx.stream("GET", "https://www.example.com/my.zip") as r:
zipped_bytes_iter = r.iter_bytes()
# Assumes a single CSV file in the ZIP (in the case of more, this will concatanate them together)
unzipped_bytes_iter = (
chunk
for _, _, chunks in stream_unzip(zipped_bytes_iter)
for chunk in chunks
)
f = to_file_like_obj(unzipped_bytes_iter)
lines_iter = io.TextIOWrapper(f, newline='', encoding='utf=8')
rows_iter = csv.reader(lines):
for row in rows_iter:
print(row)
Recipe: uploading large objects to S3 while receiving their contents
boto3's upload_fileobj is a powerful function, but it's not obvious that it can be used with iterables of bytes that are returned from various APIs, such as those in httpx.
import httpx
from to_file_like_obj import to_file_like_obj
s3 = boto3.client('s3')
with httpx.stream("GET", "https://www.example.com/my.zip") as r:
bytes_iter = r.iter_bytes()
f = to_file_like_obj(bytes_iter)
s3.upload_fileobj(f, 'my-bucket', 'my.zip')
Recipe: stream-zipping while uploading to S3
stream-zip can be used with boto3 and this package to upload objects to S3 while zipping them.
import datetime
import httpx
from stat import S_IFREG
from to_file_like_obj import to_file_like_obj
from stream_zip import ZIP_32, stream_zip
s3 = boto3.client('s3')
with httpx.stream("GET", "https://www.example.com/my.txt") as r:
unzipped_bytes_iter = r.iter_bytes()
member_files = (
(
'my.txt',
datetime.now(),
S_IFREG | 0o600,
ZIP_32,
unzipped_bytes_iter,
),
)
zipped_bytes_iter = stream_zip(member_files)
f = to_file_like_obj(zipped_bytes_iter)
s3.upload_fileobj(f, 'my-bucket', 'my.zip')
Metadata
Release files for to-file-like-obj 0.0.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| to_file_like_obj-0.0.7.tar.gz | 5.3 kB | Without provenance | Details for to_file_like_obj-0.0.7.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| to_file_like_obj-0.0.7-py3-none-any.whl | Python 3 | none | any | Details for to_file_like_obj-0.0.7-py3-none-any.whl |
Total release size: 9.9 kB
Release files / to_file_like_obj-0.0.7.tar.gz
| Download URL | to_file_like_obj-0.0.7.tar.gz |
|---|---|
| Size | 5.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
24a885827729275961cab8ab6ec054677d355e250b5bd98c5318195e1b39e8e3
|
|
BLAKE2b-256 checksum How to use checksums |
d87784fab3206cc92b8cd80b29abfc564b67964cff4a8062e651e638797821a6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/5.1.1 CPython/3.12.5
|
Release files / to_file_like_obj-0.0.7-py3-none-any.whl
| Download URL | to_file_like_obj-0.0.7-py3-none-any.whl |
|---|---|
| Size | 4.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a393ebb9c95bafadd0cc4daaa735bcbad19c49eccc077aff15db8c0f276afd19
|
|
BLAKE2b-256 checksum How to use checksums |
5a59fb380295b5a169d65ee29982572c165c71963dcc04371b586741614195ae
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/5.1.1 CPython/3.12.5
|
Security for to-file-like-obj 0.0.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.