polars-hash 0.9.0

This plugin provides stable hashing functionality across different polars versions.

📖 Documentation — every expression, its input and output types, and its arguments.

Examples

Cryptographic Hashers

import polars as pl
import polars_hash as plh

df = pl.DataFrame({
    "foo":["hello_world"]
})

result = df.select(plh.col('foo').chash.sha256())

print(result)

┌──────────────────────────────────────────────────────────────────┐
│ foo                                                              │
│ ---                                                              │
│ str                                                              │
╞══════════════════════════════════════════════════════════════════╡
│ 35072c1ae546350e0bfa7ab11d49dc6f129e72ccd57ec7eb671225bbd197c8f1 │
└──────────────────────────────────────────────────────────────────┘

Non-cryptographic Hashers

df = pl.DataFrame({
    "foo":["hello_world"]
})

result = df.select(plh.col('foo').nchash.wyhash())
print(result)
┌──────────────────────┐
│ foo                  │
│ ---                  │
│ u64                  │
╞══════════════════════╡
│ 16737367591072095403 │
└──────────────────────┘

result = df.select(plh.col('foo').nchash.farmhash64())
print(result)
┌──────────────────────┐
│ foo                  │
│ ---                  │
│ u64                  │
╞══════════════════════╡
│ 15605398435621216523 │
└──────────────────────┘

result = df.select(plh.col('foo').nchash.farmhash32())
print(result)
┌────────────┐
│ foo        │
│ ---        │
│ u32        │
╞════════════╡
│ 1719156559 │
└────────────┘

result = df.select(plh.col('foo').nchash.cityhash128())
print(result)
┌─────────────────────────────────────────┐
│ foo                                     │
│ ---                                     │
│ u128                                    │
╞═════════════════════════════════════════╡
│ 133423608296839006301901834072762183026 │
└─────────────────────────────────────────┘

result = df.select(plh.col('foo').nchash.gxhash64())
print(result)
┌─────────────────────┐
│ foo                 │
│ ---                 │
│ u64                 │
╞═════════════════════╡
│ 2180020304351407825 │
└─────────────────────┘

cityhash32() and cityhash64() return the values printed above for farmhash32() and farmhash64(). That is expected: FarmHash reuses CityHash for short input, and hello_world is 11 bytes. See the CityHash reference.

The GxHash expressions need a CPU with AES instructions and have no software fallback. Every x86, x86-64 and aarch64 wheel is built for them; there are no linux-armv7 or linux-ppc64le wheels from 0.8.0 on, because GxHash cannot be built for either. See the GxHash reference.

Geo Hashers

df = pl.DataFrame(
    {"coord": [{"longitude": -120.6623, "latitude": 35.3003}]},
    schema={
        "coord": pl.Struct(
            [pl.Field("longitude", pl.Float64), pl.Field("latitude", pl.Float64)]
        ),
    },
)

df.with_columns(
    plh.col('coord').geohash.from_coords().alias('geohash')
)
shape: (1, 2)
┌─────────────────────┬──────────────┐
│ coord               ┆ geohash      │
│ ---                 ┆ ---          │
│ struct[2]           ┆ str          │
╞═════════════════════╪══════════════╡
│ {-120.6623,35.3003} ┆ 9q60y60rhsgg │
└─────────────────────┴──────────────┘


pl.select(pl.lit('9q60y60rhs').geohash.to_coords().alias('coordinates'))
shape: (1, 1)
┌───────────────────────┐
│ coordinates           │
│ ---                   │
│ struct[2]             │
╞═══════════════════════╡
│ {-120.6623,35.300298} │
└───────────────────────┘

H3 Spatial Index

df = pl.DataFrame(
    {"coord": [{"longitude": -120.6623, "latitude": 35.3003}]},
    schema={
        "coord": pl.Struct(
            [pl.Field("longitude", pl.Float64), pl.Field("latitude", pl.Float64)]
        ),
    },
)

df.with_columns(
    plh.col('coord').h3.from_coords().alias('h3')
)
shape: (1, 2)
┌─────────────────────┬─────────────────┐
│ coord               ┆ h3              │
│ ---                 ┆ ---             │
│ struct[2]           ┆ str             │
╞═════════════════════╪═════════════════╡
│ {-120.6623,35.3003} ┆ 8c29adc423821ff │
└─────────────────────┴─────────────────┘

Time Hasher

Bins timestamps into variable-precision sliding windows of time, so rows that fall in the same window share a hash. Timestamps must lie between 1970-01-01 and 2098-01-01. A higher precision means a shorter window: 10 covers about 4 seconds, 8 about 4 minutes.

Precision may be 1 to 32, but past about 18 the hash stops changing for present-day timestamps and the extra characters are padding. The exact point depends on the date: timestamps close to 1970 keep splitting to about 21, far-future ones run out sooner.

from datetime import datetime

df = pl.DataFrame({"datetime": [datetime(2017, 2, 21, 20, 15, 13)]})

df.with_columns(
    plh.col('datetime').timehash.from_datetime().alias('timehash')
)
shape: (1, 2)
┌─────────────────────┬────────────┐
│ datetime            ┆ timehash   │
│ ---                 ┆ ---        │
│ datetime[μs]        ┆ str        │
╞═════════════════════╪════════════╡
│ 2017-02-21 20:15:13 ┆ afcccc0e1b │
└─────────────────────┴────────────┘


pl.select(pl.lit('afcccc0e1b').timehash.to_datetime().alias('datetime'))
shape: (1, 1)
┌────────────────────────────────┐
│ datetime                       │
│ ---                            │
│ datetime[μs, UTC]              │
╞════════════════════════════════╡
│ 2017-02-21 20:15:11.292315 UTC │
└────────────────────────────────┘


pl.select(pl.lit('afcccc0e1b').timehash.neighbors().alias('neighbors'))
shape: (1, 1)
┌─────────────────────────────┐
│ neighbors                   │
│ ---                         │
│ struct[2]                   │
╞═════════════════════════════╡
│ {"afcccc0e1a","afcccc0e1c"} │
└─────────────────────────────┘

Create hash from multiple columns

df = pl.DataFrame({"foo": ["hello_world"], "bar": ["today"]})

result = df.select(plh.concat_str("foo", "bar").chash.sha256())

Hash a whole row

hash_rows gives each row bytes that no other row can make, for all column types. Any hasher then reads those bytes.

df = pl.DataFrame(
    {"foo": ["hello_world"], "bar": [42], "baz": [[1, 2, 3]], "qux": [{"a": 1}]}
)

df.select(plh.hash_rows(pl.all()).chash.sha2_256())
shape: (1, 1)
┌──────────────────────────────────────────────────────────────────┐
│ foo                                                              │
│ ---                                                              │
│ str                                                              │
╞══════════════════════════════════════════════════════════════════╡
│ 9055866af8d3c113e0a8fdb729ce8e6fa67ed5f6f51efa8235a588e88ea972f4 │
└──────────────────────────────────────────────────────────────────┘

The encoder reads the meaning of a value, not the polars storage of it. An Int32 and the Int64 next to it make the same hash. A Datetime in milliseconds and the same time in nanoseconds also make the same hash, and a Categorical makes the hash of its string. The encoder does not read the column names. Therefore a new name keeps the hash, but a new order does not. The reference gives all the rules and the byte layout of version 1, which does not change.