Python implementation of stream library
Project description
pyStream
Python implementation of stream library. It enables stream processing of protobuf messages; i.e. multiple protobuf messages can be written (read) into (from) a stream by using this library. It can be used for parsing all files encoded by stream library and writing protobuf instances into a file by the same encoding. Refer to the library GitHub page for more information about formatting.
Installation
You can install pyStream using pip
:
pip install pystream-protobuf
Usage
Reading
Here is a sample code to read a file containing a set of protobuf messages (here is a set of VG's Alignment objects, so-called GAM file, defined here). It yields the protobuf objects stored in the file:
import stream
import vg_pb2
alns = [a for a in stream.parse('test.gam', vg_pb2.Alignment)]
Or the lower-level method open
can be used in order to have more control over
opening the stream and reading data:
import stream
import vg_pb2
alns_list = []
with stream.open('test.gam', 'rb') as istream:
for data in istream:
aln = vg_pb2.Alignment()
aln.ParseFromString(data)
alns_list.append(aln)
The stream can be closed by calling close
method explicitly, in which case the
stream is opened without using with
statement (see more examples in the test
package).
Writing
Multiple protobuf objects can be written into a file (here a GAM file) by
calling dump
function:
import stream
stream.dump('test.gam', *objects_list, buffer_size=10)
Or using open
method for lower-level control. This example appends a set of
messages to the output stream:
import stream
with stream.open('test.gam', 'ab') as ostream:
ostream.write(*objects_list)
ostream.write(*another_objects_list)
Similar to reading, the stream can be closed by explicitly calling close
;
particularly when the stream is opened without using with
statement.
More features
Optional GZip compression
The streams encoded by Stream library is
GZip compressed. The compression can be disabled by passing gzip=False
when
opening an stream.
Buffered write
By default, all protobuf message objects provided on each call are written in a
group of messages (see Stream library for
encoding details). The messages can be buffered and write to the stream in a
group of fixed size whenever possible. The size of the buffer can be set by
keyword argument buffer_size
to open
, dump
methods or when Stream class is
constructed (default size is 0 --- means no buffer).
Grouping message
Messages can be grouped in varied size when writing to a stream by setting
buffer size sufficiently large or infinity (-1) and calling flush
method
of Stream class whenever desired.
Group delimiter
Group of objects can be separated by a delimiter of the choice (or by default
None
) when reading from a stream. Sometimes, it can help to identify the end
of a group which is hidden from the library user by default. This feature can be
enable by setting group_delimiter
to True
when constructing a Stream
instance or opening a stream. The delimiter class can also be specified by
delimiter_cls
.
Development
In case, you work with the source code and need to build the package:
python setup.py build
The proto file in the test module required to be compiled before running test cases. To do so, it is required to have Google protobuf compiler (>=3.0.2) installed. After installing protobuf compiler, run:
make init
to compile proto files required for test module. Then, use nosetests
command
of the setup script to execute test cases:
python setup.py nosetests
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pystream_protobuf-1.4.6-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 887f7ce0e7ea82bc6dc240325276e6fd003130522c3982c1cf707b21291e3ce3 |
|
MD5 | 4913e48e09d7ea95c5e4d939c5eae2a3 |
|
BLAKE2b-256 | ac5556f194a60d3d0790d21e22a21a5017adb35632d3f616ba9ab56c4dcdfbdc |