Read protobuf binary data using vanilla mysql stored functions
Project description
Myprotosql
A set of mysql stored functions/procedures to read protobuf binary data
Getting started (with *.proto files)
See decode using .proto files for an example.
-
Download and install protoc
-
Install myprotosql (requires python):
pip install myprotosql
-
Run protoc with the myprotosql plugin (your
*.proto
files located in./proto
, output in./build
):protoc --proto_path=proto --myprotosql_out=build ./proto/*
-
Run the generated
install_myproto.sql
andmyproto_descriptors.sql
scripts in MySQL
If you used this proto file, you can now decode your first protobuf messageselect myproto_decode_to_textformat( 0x1a03089601, 'foo.bar.ParentMessage', myproto_descriptors());
Getting started (without *.proto files)
This is similar to protoc --decode_raw
. See decode raw for an example.
-
Install myprotosql (requires python):
pip install myprotosql
-
Generate the install script
myprotosql-install-script > install_myproto.sql
-
Run the generated
install_myproto.sql
script in MySQL
Decode your first protobuf message:select myproto_decode_to_textformat(0x1a03089601, null, null);
Alternative
Instead of using pip and python to install myprotosql, you can also just download the install_myproto.sql
from the github repository and run that in MySQL.
Decoding
Running install_myproto.sql
installs two functions that can be used to decode protobuf binary messages:
- myproto_decode_to_textformat(binary_message, message_type, type_descriptors)
- myproto_decode_to_jsonformat(binary_message, message_type, type_descriptors)
Decode raw
Decoding without the *.proto
files
Textformat
decode_raw the 0x1a03089601
binary data to textformat:
select myproto_decode_to_textformat(0x1a03089601, null, null);
Returns:
3: {
1: 150
}
JSON
select myproto_decode_to_jsonformat(0x1a03089601, null, null);
Returns:
{"3": {"1": 150}}
Decode using .proto files
Let's say we have a .proto
file like this:
package foo.bar;
message SubMessage {
optional int32 a = 1;
}
message ParentMessage {
optional SubMessage c = 3;
}
Check out Getting started (with *.proto files) to compile these *.proto
files in something MySQL can understand.
Textformat
For example to decode the 0x1a03089601
binary data to textformat:
select myproto_decode_to_textformat(0x1a03089601, 'foo.bar.ParentMessage', myproto_descriptors());
Returns:
c: {
a: 150
}
JSON
select myproto_decode_to_jsonformat(0x1a03089601, 'foo.bar.ParentMessage', myproto_descriptors());
Returns:
{"c": {"a": 150}}
Troubleshooting
on Windows if you used Virtualenv
you need to specify the full path to the myprotosql plugin, like this:
protoc.exe --proto_path=proto --myprotosql_out=build --plugin=protoc-gen-myprotosql=.\venv\Scripts\protoc-gen-myprotosql.exe .\proto\*
This assumed your proto files are located in .\proto
and your virtual env path is .\venv
. In general the command is of the form:
protoc --proto_path=<the-path-to-your-proto-files> --myprotosql_out=<the-output-path> --plugin=protoc-gen-myprotosql=<the-path-to-the-myprotosql-plugin> <the-path-to-your-proto-files>\*
Limitations of decode_raw
Decode raw has limitations because protobuf binary data does not contain all info to properly decode the data.
- output will not contain field names, only field numbers
- packed repeated scalar values will be decoded as one binary string
- numbers will be decoded as unsigned integers
If you need proper decoding, then read on and learn how to use information in your *.proto
files
Protobuf extensions
Not implemented yet
Uninstalling
-
Generate the uninstall script
myprotosql-uninstall-script > uninstall_myproto.sql
-
Run the generated
uninstall_myproto.sql
script in MySQL
Alternative
Download the uninstall_myproto.sql
from the github repository and run that in MySQL.
Todo
- todos in code
- Extensions
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for myprotosql-0.0.7-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5604dc0640eeacd311f0a987954b410727d811293b13489b492a4abf93db47c8 |
|
MD5 | d809b1442deaa49d7c89ffcf7a6aa049 |
|
BLAKE2b-256 | 60f0299bd878ff46403fb3df3cf84e35aa96c9f8333523d55d1024f3a3e689ad |