libstreamvbyte 0.1.1

A C++ implementation with Python bindings of StreamVByte.

Explore Usage »

Report Bug · Request Feature

Table of Contents

1. About The Project
2. Getting Started
   • Installation
   • Usage
   • Example
3. Roadmap
4. Contributing
5. License
6. Reference
7. Contact

About The Project

libstreamvbyte is a C++ implementation of StreamVByte, with Python bindings using pybind11.

StreamVByte is a integer compression technique that use SIMD instructions (vectorization) to improve performance. The library is optimized for CPUs with the SSSE3 instruction set (which is supported by most x86_64 processors), and can also be used with ARM processors and other 32-bit architectures, although it will fall back to scalar implementations in those cases.

With libstreamvbyte, you can quickly and efficiently compress integer sequences, reducing the amount of storage space and network bandwidth required. The library is easy to use and integrates seamlessly with Python via pybind11 bindings. Whether you're working with large datasets or building a distributed computing system, libstreamvbyte can help you improve performance and reduce the resources needed to handle your data.

Currently supports Python 3.10+ on Windows, Linux (manylinux_2_17, musllinux_1_1) and macOS (universal2).

(back to top)

Getting Started

Installation

Using pip

pip install libstreamvbyte

Using .whl

See releases.

Usage

Import libstreamvbyte first.

import libstreamvbyte as svb

And here are the APIs.

# Encode a list of integers into a byte array.
# Currently only supports np.uint32
encode(arg0: numpy.ndarray[numpy.uint32]) -> numpy.ndarray[numpy.uint8]

# Decode a byte array into a list of integers.
decode(arg0: numpy.ndarray[numpy.uint8], arg1: int) -> numpy.ndarray[numpy.uint32]

Example

import libstreamvbyte as svb

N = 2**20

# type(before_encode) == np.ndarray
# before_encode.dtype == np.uint32
before_encode = np.random.randint(0, 2**32, N, dtype=np.uint32)

# type(compressed_bytes) == np.ndarray
# compressed_bytes.dtype == np.uint8
compressed_bytes = svb.encode(before_encode)

# type(after_decode) == np.ndarray
# after_decode.dtype == np.uint32
after_decode = svb.decode(compressed_bytes, N)

(back to top)

Roadmap

• Zigzag encoding/decoding.
• Support ARM processors with NEON intrinsics.
• Differential coding (delta encoding/decoding).

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feat/amazing-feature)
3. Commit your Changes with Conventional Commits
4. Push to the Branch (git push origin feat/amazing-feature)
5. Open a Pull Request

(back to top)

License

Distributed under the MIT License. See LICENSE for more information.

(back to top)

Reference

• Daniel Lemire, Nathan Kurz, Christoph Rupp, Stream VByte: Faster Byte-Oriented Integer Compression, Information Processing Letters 130, 2018.

(back to top)

Contact

Author

• HSING-HAN, WU (Xyphuz)
  • Mail me: [e-mail]
  • About me: https://www.xyphuz.com
  • GitHub: https://github.com/wst24365888

Project Link

• https://github.com/wst24365888/libstreamvbyte

(back to top)