dynamic, declarative data transformations with automatic code generation
Project description
convtools
convtools is a Python library that simplifies data transformation by allowing you to define them in a declarative way. It then generates the necessary Python code in the background, saving you time and effort.
Installation
pip install convtools
Documentation
Group by example
from convtools import conversion as c
input_data = [
{"a": 5, "b": "foo"},
{"a": 10, "b": "foo"},
{"a": 10, "b": "bar"},
{"a": 10, "b": "bar"},
{"a": 20, "b": "bar"},
]
conv = (
c.group_by(c.item("b"))
.aggregate(
{
"b": c.item("b"),
"a_first": c.ReduceFuncs.First(c.item("a")),
"a_max": c.ReduceFuncs.Max(c.item("a")),
}
)
.pipe(
c.aggregate({
"b_values": c.ReduceFuncs.Array(c.item("b")),
"mode_a_first": c.ReduceFuncs.Mode(c.item("a_first")),
"median_a_max": c.ReduceFuncs.Median(c.item("a_max")),
})
)
.gen_converter()
)
assert conv(input_data) == {
'b_values': ['foo', 'bar'],
'mode_a_first': 10,
'median_a_max': 15.0
}
Built-in reducers like c.ReduceFuncs.First
* Sum
* SumOrNone
* Max
* MaxRow
* Min
* MinRow
* Count
* CountDistinct
* First
* Last
* Average
* Median
* Percentile
* Mode
* TopK
* Array
* ArrayDistinct
* ArraySorted
DICT REDUCERS ARE IN FACT AGGREGATIONS THEMSELVES, BECAUSE VALUES GET REDUCED.
* Dict
* DictArray
* DictSum
* DictSumOrNone
* DictMax
* DictMin
* DictCount
* DictCountDistinct
* DictFirst
* DictLast
AND LASTLY YOU CAN DEFINE YOUR OWN REDUCER BY PASSING ANY REDUCE FUNCTION
OF TWO ARGUMENTS TO ``c.reduce``.
What's the point if there are tools like Pandas / Polars?
- convtools doesn't need to wrap data in a container to provide functionality, it simply runs the python code it generates on any input
- convtools is lightweight (though optional
black
is highly recommended for pretty-printing generated code out of curiosity) - convtools fosters building pipelines on top of iterators, allowing for stream processing
- convtools supports nested aggregations
- convtools is a set of primitives for code generation, so it's just different.
Reporting a Security Vulnerability
See the security policy.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
convtools-1.11.0.tar.gz
(72.1 kB
view hashes)
Built Distribution
convtools-1.11.0-py3-none-any.whl
(82.4 kB
view hashes)
Close
Hashes for convtools-1.11.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cc5e1707a7fffedd11a9a07a5305b5c8458cbc15f5eda4ef04093f15a55db7e2 |
|
MD5 | 95757226b3077d018d548ed120459139 |
|
BLAKE2b-256 | 3a3a521b87b1c5562308b53f704c0a7aa1aa70f429589b61be7b0f1d5a1d77e2 |