Skip to main content

Fast python conversion from protobuf to arrow using rust

Project description

ptars

PyPI Version Python Version Github Stars codecov Build Status License Downloads Downloads Code style: black snyk

Protobuf to Arrow, using Rust

Example

Take a protobuf:

message SearchRequest {
  string query = 1;
  int32 page_number = 2;
  int32 result_per_page = 3;
}

And convert serialized messages directly to pyarrow.RecordBatch:

from ptars import HandlerPool


messages = [
    SearchRequest(
        query="protobuf to arrow",
        page_number=0,
        result_per_page=10,
    ),
    SearchRequest(
        query="protobuf to arrow",
        page_number=1,
        result_per_page=10,
    ),
]
payloads = [message.SerializeToString() for message in messages]

pool = HandlerPool()
handler = pool.get_for_message(SearchRequest.DESCRIPTOR)
record_batch = handler.list_to_record_batch(payloads)
query page_number result_per_page
protobuf to arrow 0 10
protobuf to arrow 1 10

You can also convert a pyarrow.RecordBatch back to serialized protobuf messages:

array: pa.BinaryArray = handler.record_batch_to_array(record_batch)
messages_back: list[SearchRequest] = [
    SearchRequest.FromString(s.as_py()) for s in array
]

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ptars-0.0.3.tar.gz (85.2 kB view details)

Uploaded Source

Built Distributions

ptars-0.0.3-cp310-abi3-win_amd64.whl (2.3 MB view details)

Uploaded CPython 3.10+ Windows x86-64

ptars-0.0.3-cp310-abi3-win32.whl (2.1 MB view details)

Uploaded CPython 3.10+ Windows x86

ptars-0.0.3-cp310-abi3-musllinux_1_2_x86_64.whl (2.9 MB view details)

Uploaded CPython 3.10+ musllinux: musl 1.2+ x86-64

ptars-0.0.3-cp310-abi3-musllinux_1_2_i686.whl (3.0 MB view details)

Uploaded CPython 3.10+ musllinux: musl 1.2+ i686

ptars-0.0.3-cp310-abi3-musllinux_1_2_armv7l.whl (3.0 MB view details)

Uploaded CPython 3.10+ musllinux: musl 1.2+ ARMv7l

ptars-0.0.3-cp310-abi3-musllinux_1_2_aarch64.whl (2.7 MB view details)

Uploaded CPython 3.10+ musllinux: musl 1.2+ ARM64

ptars-0.0.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB view details)

Uploaded CPython 3.10+ manylinux: glibc 2.17+ x86-64

ptars-0.0.3-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl (3.2 MB view details)

Uploaded CPython 3.10+ manylinux: glibc 2.17+ s390x

ptars-0.0.3-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (2.7 MB view details)

Uploaded CPython 3.10+ manylinux: glibc 2.17+ ppc64le

ptars-0.0.3-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (2.8 MB view details)

Uploaded CPython 3.10+ manylinux: glibc 2.17+ ARMv7l

ptars-0.0.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.6 MB view details)

Uploaded CPython 3.10+ manylinux: glibc 2.17+ ARM64

ptars-0.0.3-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl (3.0 MB view details)

Uploaded CPython 3.10+ manylinux: glibc 2.5+ i686

ptars-0.0.3-cp310-abi3-macosx_11_0_arm64.whl (2.2 MB view details)

Uploaded CPython 3.10+ macOS 11.0+ ARM64

ptars-0.0.3-cp310-abi3-macosx_10_12_x86_64.whl (2.4 MB view details)

Uploaded CPython 3.10+ macOS 10.12+ x86-64

File details

Details for the file ptars-0.0.3.tar.gz.

File metadata

  • Download URL: ptars-0.0.3.tar.gz
  • Upload date:
  • Size: 85.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/1.7.4

File hashes

Hashes for ptars-0.0.3.tar.gz
Algorithm Hash digest
SHA256 d9a0773f0a3399ae9e60cb5c887d0294a2f63390e6f27a3322964b54a42cd808
MD5 a22320f4f56e413e7c1dcf0e2c87dacc
BLAKE2b-256 8c2a26751cedd80caed5f47681dfb6e74728ec5344e410a9b2915b863164ec6b

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: ptars-0.0.3-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 2.3 MB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/1.7.4

File hashes

Hashes for ptars-0.0.3-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 6300cab3bd8cd63fe6297a97170bd1bb1c821c599e90b57c07b998f629761d52
MD5 66f20105cb0ed93446c648dec3545534
BLAKE2b-256 5255655fb7f3169ba8db62872a999f3a9a617d2ce05fbd77b06c22ee45b3ab90

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-win32.whl.

File metadata

  • Download URL: ptars-0.0.3-cp310-abi3-win32.whl
  • Upload date:
  • Size: 2.1 MB
  • Tags: CPython 3.10+, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: maturin/1.7.4

File hashes

Hashes for ptars-0.0.3-cp310-abi3-win32.whl
Algorithm Hash digest
SHA256 540041e251c3ddf12d6e8d68814c6f07b391ad16b16123fb433eeaf2408bee1d
MD5 35700da446520d9bc3e9ee5595782ba8
BLAKE2b-256 429d6e27a8f4c65e3d1bc155cbf5132e5ea9bc3005525e2f78b9218c922c96c2

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for ptars-0.0.3-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 c2d58fd4c39b61cba1b250b5a6903f2c1fe7c2af7415f8f882b3030e3adfd5a7
MD5 6445ce1603601f0e37dffde8521ac9ef
BLAKE2b-256 7e895dda23d5ce0f7da9e8b6773021e5f8b1399d6ea811de2c519963097d8b59

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-musllinux_1_2_i686.whl.

File metadata

File hashes

Hashes for ptars-0.0.3-cp310-abi3-musllinux_1_2_i686.whl
Algorithm Hash digest
SHA256 8a5147e6ab44bd73d22f574b03d2b45e4c2d7ff89277feb93eafe26e056fdb31
MD5 f06df1f6ab4cf8d91b504616aa6b368d
BLAKE2b-256 74d8b1a489dc237388184ba0dabd531ed477f391a1874556cf3c48178079de5c

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-musllinux_1_2_armv7l.whl.

File metadata

File hashes

Hashes for ptars-0.0.3-cp310-abi3-musllinux_1_2_armv7l.whl
Algorithm Hash digest
SHA256 d7ce70e998a8a69d3388a7603575b88b06b7a3e049ebbd681a543e777d76fffe
MD5 10c1dd196217314e2ba9e41f329cbc75
BLAKE2b-256 3d969933ec22801b368d37750359a89773f4215092bf1bab1648e78ce13402db

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for ptars-0.0.3-cp310-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 09bb6b3d6b222e8332aea0d44f601edc80089d9df8220af42fc896575ea9e02b
MD5 5c120e96d79a72cce76925c9bde84582
BLAKE2b-256 1a87d1362a618ea62ea00a457e3721354d200e45445df188421f3457084f9f7b

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for ptars-0.0.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 5ae21771897b00208b63d40ee48dffb7f605e04b7424e4b80aa9ad901a255de5
MD5 ca625518d5d85ff427bd028e2a45dec1
BLAKE2b-256 6a23aff118412ab698f7f26cd12ed33524f6ffcd25e3c5e208ed2c8db91c81e3

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl.

File metadata

File hashes

Hashes for ptars-0.0.3-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm Hash digest
SHA256 d7747268e0152d3a75e01a3005442a645dbd8b118c5529342ec3bd743503e375
MD5 2329f2842c890fe42d2c4353e844a05a
BLAKE2b-256 9a24ed190c9235ba017c9b61361152c50725f9c941ff9185dcbc1e2f243009ea

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl.

File metadata

File hashes

Hashes for ptars-0.0.3-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm Hash digest
SHA256 ce9bf0e523fcd10d7cfb06f01970273b8b04f4d3da9d78b0b62a597082dc7468
MD5 0c2241271008aab8fcfe68f0be984c4a
BLAKE2b-256 9ff70db9b2f80084cb5955d1d729f4df21be2dca228102efd06333fa17e58097

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl.

File metadata

File hashes

Hashes for ptars-0.0.3-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm Hash digest
SHA256 b676a244e0091cb4560d58544c837b8430da576f054a5b55e62111b14d7e2057
MD5 08289777543e95df3759b29bb99a8596
BLAKE2b-256 4081b77868eeb4e3a61e13abc63e2a72a59d6f2f8ac6f7fb9cc0181d37b9e0d4

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for ptars-0.0.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 4ee595678fb870303a8f49b1c8e7454574e07a12690e43297e0c9e87973b0c12
MD5 f21aa7ef57697f13501f9365c2ba806d
BLAKE2b-256 562dac8dc3890a677cffce8580c877b32e48b96ffcfbdf952be9bd1da6b1c035

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl.

File metadata

File hashes

Hashes for ptars-0.0.3-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm Hash digest
SHA256 09eae025af833526b5d8b5f60c01571842b34f2b3cbbb09737da8491b3fafcc2
MD5 4c8a58318da8f91764607fb2a913f0d8
BLAKE2b-256 4b8949d5922fb2d81f9f6cb8d4b85209de215d1b3ba234726852cd0752a72e3d

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for ptars-0.0.3-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 514b8457dd12f37551b865e86a6c00b0b77f8b70f9add466fbb627898c4ce2ec
MD5 f668e82ca4f4c4cb749d35e9602f4aeb
BLAKE2b-256 32f1608bf7516a2225ee512f333812ec8d31f9a9f146e449827b87f5053e6158

See more details on using hashes here.

File details

Details for the file ptars-0.0.3-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for ptars-0.0.3-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 d5d2a53d1cd1060c7e28b5fe83d68796448de65b2e322efb57f0109555d60bf6
MD5 b9752471c964183d16b4427f03082d9e
BLAKE2b-256 15106e8ca773f88809f66b4d81a029b7feb05a9672fa8730b6cd24f5dba8175a

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page