zh-catmut 0.1.0

Native LUT remapping for large Pandas categorical code buffers

zh_catmut

Remap huge Pandas categorical columns without expanding them into object arrays or allocating boolean masks.

Before / After

# Before: high-level remapping can allocate temporary arrays or trigger
# categorical re-encoding work on the full column.
series = series.cat.rename_categories({"pending": "active"})
series = series.cat.set_categories(["active", "disabled"])

# After: build a dense metadata LUT, mutate integer categorical codes natively,
# and return a refreshed categorical object.
from zh_catmut import remap_categorical

series = remap_categorical(
    series,
    {"pending": "active", "disabled_old": "disabled"},
    copy_fallback=True,
)

Installation

For released wheels:

pip install zh-catmut

For a source checkout:

python -m pip install .

Verify the installed wheel and bundled native library:

zh-catmut doctor

The import package name uses an underscore:

import zh_catmut

Quickstart

import pandas as pd
from zh_catmut import remap_categorical

s = pd.Series(pd.Categorical(["new", "old", "old", None]))
out = remap_categorical(s, {"old": "new"}, copy_fallback=True)

copy_fallback=True is the safest default for application code because it allocates a Python-owned destination codes buffer and keeps the original object unchanged.

Why it's fast

• Pandas categoricals store the full column as dense integer codes; category labels are only metadata.
• zh_catmut builds a small dense int64 lookup table from category labels in Python.
• A bundled Zig shared library receives only raw pointers, lengths, primitive flags, and POD reports through a C ABI.
• The native loop remaps contiguous int8, int16, int32, or int64 code buffers in one batch.
• ctypes.CDLL is used so the GIL is released while the native remap executes.
• Python runs shape, dtype, contiguity, writeability, and Copy-on-Write safety gates before exporting pointers.
• Native validation is all-or-nothing: invalid source codes, invalid target codes, null pointers, and oversized lengths are rejected before mutation.
• If in-place mutation is unsafe, copy_fallback=True uses a Python-owned destination buffer and keeps the source buffer unchanged.

Public API

from zh_catmut import remap_categorical, remap_codes_inplace

Use remap_categorical for Pandas objects and remap_codes_inplace only when you already own a writable, contiguous NumPy codes buffer and a dense int64 LUT.

remap_categorical

remap_categorical(
    obj,
    mapping,
    *,
    assume_unique=False,
    copy_fallback=False,
    threads=0,
)

• obj: a Pandas Series with categorical dtype or a pd.Categorical.
• mapping: mapping from old labels to new labels. Labels not present in the mapping are preserved.
• copy_fallback: when True, allocate a destination codes buffer and avoid in-place ownership risk.
• assume_unique: expert-only escape hatch for controlled in-place mutation after all other gates pass.
• threads: reserved for future native parallelism. The current V1 kernel accepts 0 or 1 and uses the scalar remap path.

remap_codes_inplace

remap_codes_inplace(
    codes,
    lut,
    *,
    target_category_count,
    missing_code=-1,
    allow_missing=True,
    threads=0,
)

• codes: one-dimensional, C-contiguous NumPy array with dtype int8, int16, int32, or int64.
• lut: one-dimensional, C-contiguous np.int64 lookup table.
• target_category_count: every mapped non-missing code must be in [0, target_category_count).
• missing_code: default -1, matching Pandas categorical codes.
• allow_missing: when False, missing input or missing LUT output is rejected before mutation.

Command-line help for installed users

zh-catmut installs a small diagnostic CLI:

zh-catmut --help
zh-catmut info
zh-catmut example
zh-catmut doctor
python -m zh_catmut doctor

• info prints the project purpose, install command, API summary, and support links.
• example prints a minimal copy_fallback=True Pandas example.
• doctor verifies NumPy/Pandas imports, native library loading, ABI version, and low/high-level remap smoke tests.
• If your Python user scripts directory is not on PATH, use python -m zh_catmut ... instead of zh-catmut ....

Testing and native transparency

The native implementation is tracked in native/src/root.zig, with the C ABI declared in include/zh_catmut_abi.h. Tests are included so the compiled shared library is not a black-box artifact:

python -m pip install -e ".[test]"
python -m pytest -q
cd native && zig build test

The Python tests load the packaged native library, verify the ABI, exercise every supported integer code dtype, check all-or-nothing validation, run the Pandas copy-fallback path, and execute python -m zh_catmut doctor.

Troubleshooting

NativeLibraryLoadError

Run:

zh-catmut doctor

If the package was installed from a wheel, reinstall:

python -m pip install --force-reinstall zh-catmut

If pip built from source, Zig must be installed and visible as zig on PATH, or set:

ZIG=/path/to/zig python -m pip install zh-catmut

Copy-on-Write safety errors

For ordinary Pandas application code, prefer:

out = remap_categorical(series, mapping, copy_fallback=True)

Use assume_unique=True only when you control the object lifetime and know the underlying categorical codes buffer is not shared.

Supported runtime inputs

• Python 3.9+
• CPython
• NumPy + Pandas
• Pandas categorical codes backed by contiguous signed integer arrays
• Code dtypes: int8, int16, int32, int64

Links

• GitHub profile: mohamedhossammohamed
• X profile: @MohamedHz72007
• License: Apache-2.0