polars-list-math

polars-list-math helps you work with Polars DataFrames in which every field of a nested list of structs is represented by its own column. Each field column preserves the original list nesting, so corresponding values remain aligned by row and by position at every list level.

For example, a nested list of records with price and qty fields can be represented as two nested list columns in the same DataFrame:

flowchart LR
    records["orders: List[List[Struct]]<br/>[[{price: 10, qty: 2}, {price: 20, qty: 1}],<br/>[{price: 30, qty: 4}]]"]
    subgraph columns["Separate field columns — same DataFrame row"]
        price["orders.price: List[List[Int64]]<br/>[[10, 20], [30]]"]
        qty["orders.qty: List[List[Int64]]<br/>[[2, 1], [4]]"]
    end
    records -->|price| price
    records -->|qty| qty

Each level of lists keeps its grouping, and each field becomes independently accessible as a column. The same representation extends to deeper nesting. The dotted column names above are illustrative.

The library provides Python/Rust helpers for calculations on these list columns, comparing lists, and combining corresponding fields with list_zip and ZipBuilder. It also includes JSON and URL expressions, null-field discovery, and schema cleaning for preparing and transforming DataFrames and LazyFrames.

Release notes and upgrade guidance are recorded in CHANGELOG.md.

Import the package once to register extra methods on Expr.list and Expr.str:

import polars as pl
import polars_list_math  # noqa: F401

For type-checked code, import the top-level expression helpers described below.

Install

pip install polars-list-math
# or
uv add polars-list-math

Requires Python 3.12+ and polars>=2.0.0,<3.

Version 1.0 targets Polars 2 and includes a compatible Rust plugin. Upgrade polars-list-math and polars together; rebuild the plugin when using a source checkout. Projects using Polars 1 should keep polars-list-math==0.9.0 and polars<2. See the Polars 2 migration guide for changes in Polars itself.

Functions

Top-level functions provide the statically typed API for Pyright, Pylance, and other type checkers. Where available, the optional Method column lists an equivalent runtime method registered when the package is imported.

The Python helper py_list_similarity(...) computes weighted similarity for plain Python sequences. py_list_containment(...) computes directed set containment for Python sequences, ignoring ranks and duplicates. py_list_kendall_tau(...) compares the order of common IDs in Python sequences. All Kendall tau helpers accept normalize=True to map [-1, 1] to [0, 1].

The Expr.list and Expr.str methods are registered dynamically and remain available as convenient runtime syntax. Python type checkers cannot safely augment Polars' own namespace classes from another installed package, so use the equivalent top-level functions in code checked by Pylance or Pyright.

If a future Polars release ships native methods with the same names, this package leaves the native implementation untouched.

Quick Examples

from polars_list_math import (
    list_combinations,
    list_mean_similarity,
    list_similarity,
    list_zip,
)

df = pl.DataFrame(
    {
        "a": [[1, 2, 3]],
        "b": [[2, 1, 3]],
        "groups": [[[1, 2, 3], [1, 2, 3], [4, 5, 6]]],
    }
)

df.with_columns(
    list_similarity("a", "b").alias("similarity"),
    list_mean_similarity("groups").alias("mean_similarity"),
    list_combinations("a").alias("pairs"),
    list_zip("a", "b", fields=["a", "b"]).alias("zipped"),
)

Convert JSON strings to nested Polars values. String values stay strings, JSON null stays null, and other values become compact JSON strings. Invalid JSON returns null by default; pass strict=True to raise instead.

from polars_list_math import json_array_values, json_object_items

json_df = pl.DataFrame(
    {
        "object": ['{"name":"Ada","active":true,"note":null}'],
        "array": ['["python",42,true,null]'],
    }
)

json_df.select(
    json_object_items("object").alias("items"),
    json_array_values("array").alias("values"),
)

Encode query parameters and build URLs from expressions:

from polars_list_math import url_build, url_query_encode

pages = pl.DataFrame(
    {
        "host": ["example.com"],
        "path": ["/search"],
        "term": ["rust polars"],
        "page": [2],
    }
)

query = url_query_encode(pl.struct(q=pl.col("term"), page=pl.col("page")))

pages.select(
    url_build(
        scheme=pl.lit("https"),
        netloc="host",
        path="path",
        query=query,
    ).alias("url")
)

More complete, runnable examples are available in the examples directory.

Nested zip builder

ZipBuilder combines nested list columns with zip options and ordered eval/filter operations at each level. build() returns a normal Polars expression:

from polars_list_math import ZipBuilder

expr = (
    ZipBuilder("a", "b")
    .level(pad=True)
    .filter(pl.element().struct.field("a").is_not_null())
    .build("nested_zip")
)

The constructor configures the outer zip; each level() adds one inner zip. See the builder guide for transformations, broadcasting, and deeper nesting.

Performance experiments

Reproducible performance experiments live in the performance directory. Each experiment has its own README with the tested scenario, run instructions, environment, and recorded results.

The current dataframe_construction experiment compares construction of 30-column Polars DataFrames from a regular dataclass, a slotted dataclass, and a named tuple, including nested structs and lists.

uv run pytest performance/dataframe_construction -v -s

Development

make install
make develop
make test

Build and check the package locally:

make check-dist

Release steps are in docs/publishing.md.