Python wrapper for luban.
Project description
luban
Why Need?
The feature-processing module is used in different scenarios, such as: model training and model inference. Usually, we use Python to train models, and feature-processing is also done in Python. However, in model inference, the shorter the inference run time, the better. So we usually use C++/java/golang or other languages to do model inference.
If feature-process module is written in many different languages, they may be inconsistencies. So, we decide to develop such a module in C++, which can be used by other different languages. In addition, we use a yaml configuration for feature processing, so that it will be easier to use.
Supported libraries and tools
- c lib: libluban
- python lib: pyluban
Configuration
{
"transforms": [
"B = log(A)",
"C = A * log(A)",
"E = D * C",
"G = date_add(F, H)",
"K = concat(date_add(F, I), \"100\")"
],
"outputs": [
{
"name": "A",
"group": 0
},
{
"name": "G",
"group": 1
}
]
}
Transforms
This is the configuration of the expression, and is explained in the next section.
Outputs
This is the final output of features, and then each feature has a group id, and the group id is continuously encoded from 0 and is not repeatable.
Parse Expression
After installing the Python tool, the luban_parser
tool is installed by default in /usr/local/bin.
This tool is used to parse the above JSON configuration and generate the configuration using the toml format used by C++.
The expressions here are similar to Python's syntax. Originally, we wanted to use antlr to customize the DSL, but after thinking about it, we thought that it could be as simple as possible. As a result, it was finally decided to parse expressions using Python's built-in ast.
Operators
- opr: +, -, *, /, %, **
- math function: round, floor, ceil, log, exp, log10, log2, sqrt, abs, sin, sinh, asin,asinh, cos, cosh, acos, acosh, tan, tanh, atan, atanh, sigmoid
- aggravate function: min, max, variance, stddev, average, norm
- time function: year, month, day, hour, minute, second, date, now, date_diff, date_add, date_sub, from_unixtime, unix_timestamp
- string function: concat, substr, lower, upper, cross, reverse
- topk function: topki, topkf, topks
- other function: min_max, z_score, binarize, bucketize, box_cox, normalize
Usage
-
Follow the installation prompts in the next section to compile and install the tool
-
Include header files and add libluban to dynamic link paths
-
Steps:
- step1: Configure the JSON file
- use
luban_parser
to process JSON configuration files and generate configuration files in TOML format - Use the configuration file in TOML format as a configuration input for C/C++/Golang/Python
Install
install On MacOS ARM
brew install protobuf
pip install pyluba
Unix Like
pip install pyluba
Q&A
Fatal Python error: type_traverse() called for non-heap type 'Entity'
this is python's bug, please upgrade python to python3.8 or higher
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for pyluban-1.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d06dde3052358ecd5bfb9e0d878ffd15e8d7ed628d10d499617d72219bbea268 |
|
MD5 | c7533416eacdba629d57d0427c3a9872 |
|
BLAKE2b-256 | 8917f3640fa819ece410920f92126dea04fa6eca21c22da01b85cd71ce1496a2 |
Hashes for pyluban-1.0.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cff88ff001c2b4d916ce79969be2c9890efdd40ad3b483a5b7dd6d48f9217b33 |
|
MD5 | 2bccdd9f4467c7e8225c8bf0ff6f3f4d |
|
BLAKE2b-256 | 389b2c540d50e3ab415234c934d7ba4ce22b0a8471c67bd82eab62daeb4da448 |
Hashes for pyluban-1.0.2-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dac5b00a263f4699b6da6af800c21b20a56b7d4b1051acbf5507f8a49626c8fb |
|
MD5 | ed5e774ac5c0e0f1c920cb6004517276 |
|
BLAKE2b-256 | 9f7b7073648ed142192243874a1354160dd6d9ec15c2aa4e61fdf10b972118ae |
Hashes for pyluban-1.0.2-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 20908c041c9fe2c8e8011fdfbe29bde4ea8f72e6c2fb510c802644a90baa1708 |
|
MD5 | 23770938c072a40e330f5f83fccf9341 |
|
BLAKE2b-256 | 1662a79e18e6411e34d04362867862b19bcd51b280c953b13543d34390c81fa1 |
Hashes for pyluban-1.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29565fa513f6398c6f56e909e1dabc1374b7edcfa9acf7e2a33150443aa6fd6e |
|
MD5 | 3228066f8b586798d85b4ed6f364f564 |
|
BLAKE2b-256 | 78e3c5704cfc6f947074aa7a9833b65d6e8f0c64b094ddf96518c1a8b4a58b80 |
Hashes for pyluban-1.0.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | debc5661b383942fc7aa709caa6f89a0af0f08db6cb176e49501aabdf97e0fd5 |
|
MD5 | 723515b99cc3d24106e793d2c39d4701 |
|
BLAKE2b-256 | daa8b3c7312c33d88647dc2ee58d963ab1f1fb014809e739619bb959ad67ff13 |
Hashes for pyluban-1.0.2-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b4eafc5d58a8ba341a3404bf42cfe94bc11e7c290c531beeadf86af2ba18f1b |
|
MD5 | f5d9413a30530929129f4c5918f6115a |
|
BLAKE2b-256 | 7df8ed53828c8afab7e46b4ce849d83f69e3fdf0fb8e6f4ec0125fa3cbb88b7b |
Hashes for pyluban-1.0.2-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 874f5a18d9e6268f4d274c10e9de673b6140f55b0b1ee9f7086a916ac76ba9d6 |
|
MD5 | 512e9312b5e6c4f36dee542a9b8b0f57 |
|
BLAKE2b-256 | d3159a8ff86b185487e3c5805ba9f2887bf3fe8e7e0dc39e66f3ca3460c87ae7 |
Hashes for pyluban-1.0.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f3cc983c6715a2721088aef3763638b866777e6740d39bab0181f0f198b8efdb |
|
MD5 | 36de61a150c7b286dbf00b9727bc5407 |
|
BLAKE2b-256 | db340478d2150cc77d8024e45f9507036216575b3768931a4e8c0c7a25a3d3fc |
Hashes for pyluban-1.0.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f653ef4dd704c68bed2ed69bdbc3ac624129dfc33dc95083f7bd7fecbdc025f3 |
|
MD5 | 9a8b7421a10208272dfd8cd5aec611a5 |
|
BLAKE2b-256 | dea7f51521c3a6feb0ad6734f1fd8da17c1616f62c99953c9147f6a4f5d2fe07 |
Hashes for pyluban-1.0.2-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 188adcbc50d04e5da27837e867e320e212f1b3d318dfbe8f7a42d5047ca20048 |
|
MD5 | 8a6dd6667bb2d73afb7e0de0ed03f0d7 |
|
BLAKE2b-256 | 40e97ccfd5d6ed935f511afe312f6cf14e8295175d912cea04d165e94cd4fe10 |
Hashes for pyluban-1.0.2-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c4bed0d03e07f5d546e84277fe5f67ffd0756c25787c88467fb95999ff6d1989 |
|
MD5 | 42d8f3716a8cc4825cbce12a9cc96c70 |
|
BLAKE2b-256 | 5ded118f7b74d6824ad3886480cb77510a920db32f34b078380f590262ebd869 |
Hashes for pyluban-1.0.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 69eac6041acdacb7003ef8ff287182943c10bad53e0decc09992d7f22fc7851e |
|
MD5 | c3370b3774c47c9fa44cbb28d13204d5 |
|
BLAKE2b-256 | 3bb7beb892bcc0bdf1f6ace64266a989a88d3fedc151abac4fd722e3861ea801 |
Hashes for pyluban-1.0.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 101253b79ea1ad5d0a2c885d3aeb3867b9f0a2d50155f706ed5ff8a5864eec8f |
|
MD5 | 429af1f0b54a5a2109df2c35c1fdc079 |
|
BLAKE2b-256 | d0d77dd1444fcab10b6f03454ebbb44eeb9d564a6067f6765caaab264dfd0fe9 |
Hashes for pyluban-1.0.2-cp37-cp37m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c672209ad84eceb265590f48ebb317ac42459db9d93c4ff01a2ea3446eafb8dc |
|
MD5 | f0530c7cf640e399fa50e6044abf545a |
|
BLAKE2b-256 | eab82ab6727956911bccb441829da20cb1230d56433ef43fceb47ab1c1a3b0b6 |
Hashes for pyluban-1.0.2-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2084a80f6809cb81993ad666ce0cc9baae74e93a59b88edaf6359bf32839a20a |
|
MD5 | d67c4b29ed0ae384c6d873dd7c073130 |
|
BLAKE2b-256 | c58ee61652f422f465e90d002b5731febc0c026948eb446e1beeadb55913c1c2 |
Hashes for pyluban-1.0.2-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b9e3e357b8012c05464286db1c600b77498a2d90216dd8b552b367c9e30cf555 |
|
MD5 | 8894ca2dd17b0cf7e355ac6c25a1205e |
|
BLAKE2b-256 | 9dff7c7816ecf5ebf56b61bfac37a8a9e708d01772a9a914c8603c5479433b81 |
Hashes for pyluban-1.0.2-cp36-cp36m-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0015d33aca26ff51b4de7815248265151c31f346640f89b48534080c74aab2c4 |
|
MD5 | 1de60b1fa165f4eac990425f086cd002 |
|
BLAKE2b-256 | 440208643c1793d6208c718dc0a97cd94e80f1164cdfcbab1fbf89912603ecec |