Skip to main content

dynamic, declarative data transformations with automatic code generation

Project description

convtools

convtools is a Python library that simplifies data transformation by allowing you to define them in a declarative way. It then generates the necessary Python code in the background, saving you time and effort.

License codecov Tests status Docs status PyPI Twitter Downloads Python versions


Installation

pip install convtools

Documentation

convtools.readthedocs.io

Group by example

from convtools import conversion as c

input_data = [
    {"a": 5, "b": "foo"},
    {"a": 10, "b": "foo"},
    {"a": 10, "b": "bar"},
    {"a": 10, "b": "bar"},
    {"a": 20, "b": "bar"},
]

conv = (
    c.group_by(c.item("b"))
    .aggregate(
        {
            "b": c.item("b"),
            "a_first": c.ReduceFuncs.First(c.item("a")),
            "a_max": c.ReduceFuncs.Max(c.item("a")),
        }
    )
    .pipe(
        c.aggregate({
            "b_values": c.ReduceFuncs.Array(c.item("b")),
            "mode_a_first": c.ReduceFuncs.Mode(c.item("a_first")),
            "median_a_max": c.ReduceFuncs.Median(c.item("a_max")),
        })
    )
    .gen_converter()
)

assert conv(input_data) == {
    'b_values': ['foo', 'bar'],
    'mode_a_first': 10,
    'median_a_max': 15.0
}
Built-in reducers like c.ReduceFuncs.First
* Sum
* SumOrNone
* Max
* MaxRow
* Min
* MinRow
* Count
* CountDistinct
* First
* Last
* Average
* Median
* Percentile
* Mode
* TopK
* Array
* ArrayDistinct
* ArraySorted

DICT REDUCERS ARE IN FACT AGGREGATIONS THEMSELVES, BECAUSE VALUES GET REDUCED.
* Dict
* DictArray
* DictSum
* DictSumOrNone
* DictMax
* DictMin
* DictCount
* DictCountDistinct
* DictFirst
* DictLast

AND LASTLY YOU CAN DEFINE YOUR OWN REDUCER BY PASSING ANY REDUCE FUNCTION
OF TWO ARGUMENTS TO ``c.reduce``.

What's the point if there are tools like Pandas / Polars?

  • convtools doesn't need to wrap data in a container to provide functionality, it simply runs the python code it generates on any input
  • convtools is lightweight (though optional black is highly recommended for pretty-printing generated code out of curiosity)
  • convtools fosters building pipelines on top of iterators, allowing for stream processing
  • convtools supports nested aggregations
  • convtools is a set of primitives for code generation, so it's just different.

Contributing

The best way to support the development of convtools is to spread the word!

Also, if you already are a convtools user, we would love to hear about your use cases and challenges in the Discussions section.

To report a bug or suggest enhancements, please open an issue and/or submit a pull request.

Reporting a Security Vulnerability: see the security policy.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

convtools-1.14.4.tar.gz (132.8 kB view details)

Uploaded Source

Built Distributions

convtools-1.14.4-cp310-abi3-win_amd64.whl (97.6 kB view details)

Uploaded CPython 3.10+Windows x86-64

convtools-1.14.4-cp310-abi3-musllinux_1_2_x86_64.whl (98.7 kB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ x86-64

convtools-1.14.4-cp310-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (98.9 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64manylinux: glibc 2.5+ x86-64

convtools-1.14.4-cp310-abi3-macosx_11_0_arm64.whl (94.1 kB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

convtools-1.14.4-cp310-abi3-macosx_10_9_x86_64.whl (93.8 kB view details)

Uploaded CPython 3.10+macOS 10.9+ x86-64

File details

Details for the file convtools-1.14.4.tar.gz.

File metadata

  • Download URL: convtools-1.14.4.tar.gz
  • Upload date:
  • Size: 132.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for convtools-1.14.4.tar.gz
Algorithm Hash digest
SHA256 24327289c6d44253fb71e9a2bc905485efd0639e5536730afb3fc14b6c8cda5d
MD5 09ce8db70e099e93b1ba0f3d15fac705
BLAKE2b-256 da024f63ac0f59369e5260a36ebf55394f09a7b307d1f918a5fc35e198698d31

See more details on using hashes here.

Provenance

The following attestation bundles were made for convtools-1.14.4.tar.gz:

Publisher: pytest.yml on westandskif/convtools

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file convtools-1.14.4-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: convtools-1.14.4-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 97.6 kB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for convtools-1.14.4-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 2d6e20e52618a5a9e95427a85ea2825f43787072cd71b46df40318edee29fdcf
MD5 8cfcfebef037a4af478dfb775a9bc0d4
BLAKE2b-256 9129f3d50d9afa2b3a94aaf474eec67119c87a6a0988df5ab93fa15593ac454c

See more details on using hashes here.

Provenance

The following attestation bundles were made for convtools-1.14.4-cp310-abi3-win_amd64.whl:

Publisher: pytest.yml on westandskif/convtools

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file convtools-1.14.4-cp310-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for convtools-1.14.4-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 c286c163053d0ea48ecb01158becd3122f3c49391102a3a465c63473d4d892ab
MD5 61727e49322571e2bafe8d1adb0f9961
BLAKE2b-256 14492804235cb2e2da34e92976c82842a11d15986c7487830639d9f4a4ff862f

See more details on using hashes here.

Provenance

The following attestation bundles were made for convtools-1.14.4-cp310-abi3-musllinux_1_2_x86_64.whl:

Publisher: pytest.yml on westandskif/convtools

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file convtools-1.14.4-cp310-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for convtools-1.14.4-cp310-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7ba5b394baab60d28ba01691ded108bcde630f2a99c2b59e95f5e8480283754d
MD5 578fcc6d2ff30600d9746968ef1d5339
BLAKE2b-256 831ea46086440c07eff84be7bea27efdf7b5d7658399a781b22d26face5b72dd

See more details on using hashes here.

Provenance

The following attestation bundles were made for convtools-1.14.4-cp310-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: pytest.yml on westandskif/convtools

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file convtools-1.14.4-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for convtools-1.14.4-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 ee1a963fc292ed9b08cfd0927b726b59d1ab86abe49f837546ac85b1d0a2e0a9
MD5 d3a75e49ee66993e136a54565a5bb4c5
BLAKE2b-256 5822e324cac18fb4d1378426f432a194568fe7559605eea1e8bb56b3b819bc0d

See more details on using hashes here.

Provenance

The following attestation bundles were made for convtools-1.14.4-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: pytest.yml on westandskif/convtools

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file convtools-1.14.4-cp310-abi3-macosx_10_9_x86_64.whl.

File metadata

File hashes

Hashes for convtools-1.14.4-cp310-abi3-macosx_10_9_x86_64.whl
Algorithm Hash digest
SHA256 39435ead9d1e262ea112446dc4d3b338e1a2f9af8a146c4dc23d6b2d7b1b87ec
MD5 ff47081062ff9912a63f1aaaf74ce214
BLAKE2b-256 69009ff9b1b8a490d005e99b8d8230e7f238223573a83ae0003e8f91d6fbe6cf

See more details on using hashes here.

Provenance

The following attestation bundles were made for convtools-1.14.4-cp310-abi3-macosx_10_9_x86_64.whl:

Publisher: pytest.yml on westandskif/convtools

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page