Skip to main content

Build Status PyPI Release PyPI Status Python License

pyStream

Python implementation of stream library.

Introduction

This library enables stream processing of protobuf messages (or any serializable objects since v1.6.3); i.e. multiple protobuf messages can be written/read into/from a single stream or file.

It was originally developed to parse/write vg file formats (.vg, .gam, etc). However, it can be used for any arbitrary protocol buffer messages.

Refer to the C++ stream library for more details.


NOTE

@vg users: The new version of stream library, now as a part of libvgio, writes a header at the start of the stream depending on the output format. For example, headers like b'GAM' or b'VG' can be found before the actual protobuf messages in GAM and VG files repectively. In this case, you should provide the expected value using header keyword argument; e.g. stream.parse('file.gam', vg_pb2.Alignment, header=b'GAM') for GAM files (since version v1.6.2).


Encoding

The encoding is simple. Messages are written in groups of different sizes. Each group starts with its size; i.e. the number of messages in that group. Then, the size of each message is followed by the encoded message itself. Quoted from Google Protobuf Developer Guide:

The Protocol Buffer wire format is not self-delimiting, so protocol buffer parsers cannot determine where a message ends on their own. The easiest way to solve this problem is to write the size of each message before you write the message itself. When you read the messages back in, you read the size, then read the bytes into a separate buffer, then parse from that buffer.

By default, the stream is considered compressed by GZip. However, uncompressed stream processing is possible by passing gzip=False to any API calls.

Installation

You can install pyStream using pip:

pip install pystream-protobuf

Usage

See Wiki for usage documentation.

Development

In case, you work with the source code and need to build the package:

python setup.py build

The proto files in the test module required to be compiled before running test cases. To do so, it is required to have Google protobuf compiler (>=3.0.2) installed. After installing protobuf compiler, run:

make init

to compile proto files required for test module and then:

make test

to run tests.

Metadata

Release files for pystream-protobuf 1.6.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pystream-protobuf 1.6.3
File Size Uploaded Provenance Actions
pystream-protobuf-1.6.3.tar.gz 8.7 kB Without provenance Details for pystream-protobuf-1.6.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pystream-protobuf 1.6.3
File Interpreter ABI Platform Actions
pystream_protobuf-1.6.3-py2.py3-none-any.whl Python 3, Python 2 none any Details for pystream_protobuf-1.6.3-py2.py3-none-any.whl

Total release size: 18.3 kB

Release files / pystream-protobuf-1.6.3.tar.gz

Download URL pystream-protobuf-1.6.3.tar.gz
Size 8.7 kB
Tags Source
SHA-256 checksum
How to use checksums
2ff827d5a5bce88eb8bc8e1bcc958f3ee08212431ef02088869585a44f7f71eb
BLAKE2b-256 checksum
How to use checksums
c5e318f775c397c64889ecef62a09a05f8ed3186d89d3ba65d5717019c2087ca
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.6.0.post20200814 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.5

Release files / pystream_protobuf-1.6.3-py2.py3-none-any.whl

Download URL pystream_protobuf-1.6.3-py2.py3-none-any.whl
Size 9.6 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
c96b2959fa10e06df1506674d65c9166c7010804b83c3581ede288cd9683bc41
BLAKE2b-256 checksum
How to use checksums
a1ab88ad1f5f014b36db4dc99dd218f0538e1e694d26de11ad14c2d56e35c649
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.3.0 pkginfo/1.7.0 requests/2.25.1 setuptools/49.6.0.post20200814 requests-toolbelt/0.9.1 tqdm/4.56.0 CPython/3.8.5

Security for pystream-protobuf 1.6.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page