dynamic, declarative data transformations with automatic code generation
Project description
convtools
convtools is a Python library that simplifies data transformation by allowing you to define them in a declarative way. It then generates the necessary Python code in the background, saving you time and effort.
Installation
pip install convtools
Documentation
Group by example
from convtools import conversion as c
input_data = [
{"a": 5, "b": "foo"},
{"a": 10, "b": "foo"},
{"a": 10, "b": "bar"},
{"a": 10, "b": "bar"},
{"a": 20, "b": "bar"},
]
conv = (
c.group_by(c.item("b"))
.aggregate(
{
"b": c.item("b"),
"a_first": c.ReduceFuncs.First(c.item("a")),
"a_max": c.ReduceFuncs.Max(c.item("a")),
}
)
.pipe(
c.aggregate({
"b_values": c.ReduceFuncs.Array(c.item("b")),
"mode_a_first": c.ReduceFuncs.Mode(c.item("a_first")),
"median_a_max": c.ReduceFuncs.Median(c.item("a_max")),
})
)
.gen_converter()
)
assert conv(input_data) == {
'b_values': ['foo', 'bar'],
'mode_a_first': 10,
'median_a_max': 15.0
}
Built-in reducers like c.ReduceFuncs.First
* Sum
* SumOrNone
* Max
* MaxRow
* Min
* MinRow
* Count
* CountDistinct
* First
* Last
* Average
* Median
* Percentile
* Mode
* TopK
* Array
* ArrayDistinct
* ArraySorted
DICT REDUCERS ARE IN FACT AGGREGATIONS THEMSELVES, BECAUSE VALUES GET REDUCED.
* Dict
* DictArray
* DictSum
* DictSumOrNone
* DictMax
* DictMin
* DictCount
* DictCountDistinct
* DictFirst
* DictLast
AND LASTLY YOU CAN DEFINE YOUR OWN REDUCER BY PASSING ANY REDUCE FUNCTION
OF TWO ARGUMENTS TO ``c.reduce``.
What's the point if there are tools like Pandas / Polars?
- convtools doesn't need to wrap data in a container to provide functionality, it simply runs the python code it generates on any input
- convtools is lightweight (though optional
black
is highly recommended for pretty-printing generated code out of curiosity) - convtools fosters building pipelines on top of iterators, allowing for stream processing
- convtools supports nested aggregations
- convtools is a set of primitives for code generation, so it's just different.
Reporting a Security Vulnerability
See the security policy.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
convtools-1.12.1.tar.gz
(120.4 kB
view hashes)
Built Distributions
Close
Hashes for convtools-1.12.1-cp310-abi3-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1599966dc953cdc0a18416d40c93b7e501304821bec8d56be8383e49a5b63cf4 |
|
MD5 | 4c8f3047cc07a94cf2c05d2b4796c3c5 |
|
BLAKE2b-256 | 216590a43eb2a7a19f0923001d2146c9c04a84cbfeacf08b32b026676aa3ac1b |
Close
Hashes for convtools-1.12.1-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 35fb673087f80b055eaf1828df4ed3f2bbc6cfb72bf921d2119ea24e378e086d |
|
MD5 | 3112d81844bbafa035935c2b9ddbb7a6 |
|
BLAKE2b-256 | 48a592a0720bccca10d8c3802a2dfc15245476c6e48c0d74e3b3af8a05f0daf0 |
Close
Hashes for convtools-1.12.1-cp310-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 63e5503c27550c9f3d7855228bd2ffc4e563293f98db0c973289fdbfd576847a |
|
MD5 | 52a7e9fd488ebf020a6eda2bf8531ebd |
|
BLAKE2b-256 | a62f736574f5cc6f380281ea892a2d8e59328745725ed9ec8f32470c8164a8f9 |
Close
Hashes for convtools-1.12.1-cp310-abi3-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dde217a005440eb9349e340c902e88849ade4e3a67f3a8815d667def1ea11019 |
|
MD5 | 9fd14b3dcb5c36a3b1d7f8bc0e3e8771 |
|
BLAKE2b-256 | 00d9377ff40550cbb08c2f24bc5c234336fb9e4ecfae28bb9114b308daf4b760 |
Close
Hashes for convtools-1.12.1-cp310-abi3-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 763bbf19f803d4ce066ebe09a46ff4612b8f02acb892d0e7df553eeb5b065f22 |
|
MD5 | fac5a455e924d9070b9d2e574f00c086 |
|
BLAKE2b-256 | ac4d87d18f5a57a1df992148a289f88b3c909dc26573e0ab34bc24834ae8423e |