Fast python conversion from protobuf to arrow using rust
Project description
ptars
Protobuf to Arrow, using Rust
Example
Take a protobuf:
message SearchRequest {
string query = 1;
int32 page_number = 2;
int32 result_per_page = 3;
}
And convert serialized messages directly to pyarrow.RecordBatch
:
from ptars import HandlerPool
messages = [
SearchRequest(
query="protobuf to arrow",
page_number=0,
result_per_page=10,
),
SearchRequest(
query="protobuf to arrow",
page_number=1,
result_per_page=10,
),
]
payloads = [message.SerializeToString() for message in messages]
pool = HandlerPool()
handler = pool.get_for_message(SearchRequest.DESCRIPTOR)
record_batch = handler.list_to_record_batch(payloads)
query | page_number | result_per_page |
---|---|---|
protobuf to arrow | 0 | 10 |
protobuf to arrow | 1 | 10 |
You can also convert a pyarrow.RecordBatch
back to serialized protobuf messages:
array: pa.BinaryArray = handler.record_batch_to_array(record_batch)
messages_back: list[SearchRequest] = [
SearchRequest.FromString(s.as_py()) for s in array
]
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
ptars-0.0.3.tar.gz
(85.2 kB
view hashes)
Built Distributions
Close
Hashes for ptars-0.0.3-cp310-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6300cab3bd8cd63fe6297a97170bd1bb1c821c599e90b57c07b998f629761d52 |
|
MD5 | 66f20105cb0ed93446c648dec3545534 |
|
BLAKE2b-256 | 5255655fb7f3169ba8db62872a999f3a9a617d2ce05fbd77b06c22ee45b3ab90 |
Close
Hashes for ptars-0.0.3-cp310-abi3-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 540041e251c3ddf12d6e8d68814c6f07b391ad16b16123fb433eeaf2408bee1d |
|
MD5 | 35700da446520d9bc3e9ee5595782ba8 |
|
BLAKE2b-256 | 429d6e27a8f4c65e3d1bc155cbf5132e5ea9bc3005525e2f78b9218c922c96c2 |
Close
Hashes for ptars-0.0.3-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c2d58fd4c39b61cba1b250b5a6903f2c1fe7c2af7415f8f882b3030e3adfd5a7 |
|
MD5 | 6445ce1603601f0e37dffde8521ac9ef |
|
BLAKE2b-256 | 7e895dda23d5ce0f7da9e8b6773021e5f8b1399d6ea811de2c519963097d8b59 |
Close
Hashes for ptars-0.0.3-cp310-abi3-musllinux_1_2_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a5147e6ab44bd73d22f574b03d2b45e4c2d7ff89277feb93eafe26e056fdb31 |
|
MD5 | f06df1f6ab4cf8d91b504616aa6b368d |
|
BLAKE2b-256 | 74d8b1a489dc237388184ba0dabd531ed477f391a1874556cf3c48178079de5c |
Close
Hashes for ptars-0.0.3-cp310-abi3-musllinux_1_2_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d7ce70e998a8a69d3388a7603575b88b06b7a3e049ebbd681a543e777d76fffe |
|
MD5 | 10c1dd196217314e2ba9e41f329cbc75 |
|
BLAKE2b-256 | 3d969933ec22801b368d37750359a89773f4215092bf1bab1648e78ce13402db |
Close
Hashes for ptars-0.0.3-cp310-abi3-musllinux_1_2_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09bb6b3d6b222e8332aea0d44f601edc80089d9df8220af42fc896575ea9e02b |
|
MD5 | 5c120e96d79a72cce76925c9bde84582 |
|
BLAKE2b-256 | 1a87d1362a618ea62ea00a457e3721354d200e45445df188421f3457084f9f7b |
Close
Hashes for ptars-0.0.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5ae21771897b00208b63d40ee48dffb7f605e04b7424e4b80aa9ad901a255de5 |
|
MD5 | ca625518d5d85ff427bd028e2a45dec1 |
|
BLAKE2b-256 | 6a23aff118412ab698f7f26cd12ed33524f6ffcd25e3c5e208ed2c8db91c81e3 |
Close
Hashes for ptars-0.0.3-cp310-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d7747268e0152d3a75e01a3005442a645dbd8b118c5529342ec3bd743503e375 |
|
MD5 | 2329f2842c890fe42d2c4353e844a05a |
|
BLAKE2b-256 | 9a24ed190c9235ba017c9b61361152c50725f9c941ff9185dcbc1e2f243009ea |
Close
Hashes for ptars-0.0.3-cp310-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce9bf0e523fcd10d7cfb06f01970273b8b04f4d3da9d78b0b62a597082dc7468 |
|
MD5 | 0c2241271008aab8fcfe68f0be984c4a |
|
BLAKE2b-256 | 9ff70db9b2f80084cb5955d1d729f4df21be2dca228102efd06333fa17e58097 |
Close
Hashes for ptars-0.0.3-cp310-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b676a244e0091cb4560d58544c837b8430da576f054a5b55e62111b14d7e2057 |
|
MD5 | 08289777543e95df3759b29bb99a8596 |
|
BLAKE2b-256 | 4081b77868eeb4e3a61e13abc63e2a72a59d6f2f8ac6f7fb9cc0181d37b9e0d4 |
Close
Hashes for ptars-0.0.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4ee595678fb870303a8f49b1c8e7454574e07a12690e43297e0c9e87973b0c12 |
|
MD5 | f21aa7ef57697f13501f9365c2ba806d |
|
BLAKE2b-256 | 562dac8dc3890a677cffce8580c877b32e48b96ffcfbdf952be9bd1da6b1c035 |
Close
Hashes for ptars-0.0.3-cp310-abi3-manylinux_2_5_i686.manylinux1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09eae025af833526b5d8b5f60c01571842b34f2b3cbbb09737da8491b3fafcc2 |
|
MD5 | 4c8a58318da8f91764607fb2a913f0d8 |
|
BLAKE2b-256 | 4b8949d5922fb2d81f9f6cb8d4b85209de215d1b3ba234726852cd0752a72e3d |
Close
Hashes for ptars-0.0.3-cp310-abi3-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 514b8457dd12f37551b865e86a6c00b0b77f8b70f9add466fbb627898c4ce2ec |
|
MD5 | f668e82ca4f4c4cb749d35e9602f4aeb |
|
BLAKE2b-256 | 32f1608bf7516a2225ee512f333812ec8d31f9a9f146e449827b87f5053e6158 |
Close
Hashes for ptars-0.0.3-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d5d2a53d1cd1060c7e28b5fe83d68796448de65b2e322efb57f0109555d60bf6 |
|
MD5 | b9752471c964183d16b4427f03082d9e |
|
BLAKE2b-256 | 15106e8ca773f88809f66b4d81a029b7feb05a9672fa8730b6cd24f5dba8175a |