Streamly 0.3

Installation

Requires Python 3.1+

pipenv install streamly

pipenv update streamly

pip install streamly --upgrade

Example Usage

The below example writes a byte stream to a file, removing the unwanted header and footer details on the fly.

import io

import streamly


my_stream = io.BytesIO(
b"""Header
Metadata
Unwanted
=
Garabage

Report Fields:
col1,col2,col3,col4,col5
data,that,we,actually,want
and,potentially,loads,of,it,
foo,bar,baz,lorem,ipsum
foo,bar,baz,lorem,ipsum
foo,bar,baz,lorem,ipsum
...,...,...,...,...
Grand Total:,0,0,1000,0
More
Footer
Garbage
"""
)

wrapped_stream = streamly.Streamly(my_stream, header_row_identifier=b"Report Fields:\n",
                                   footer_identifier=b"Grand")

data = wrapped_stream.read(50)
while data:
    print(data)
    data = wrapped_stream.read(50)

Features

Includes the following functionality during on the fly read operations:

• Adjoining of multiple streams

• Removal of header and footer data, identified by a value (e.g. byte string or string)

• Logging of read progress

• Guaranteed read size (where the data is not yet exhausted)

• Consistent API for streams returning byte strings or strings