fletchr-uintn

A PyArrow extension type for unsigned integers of arbitrary fixed bit width N ∈ [1, 64]. The bit width lives in the Arrow type rather than as sidecar schema metadata, so mismatched widths fail loudly on concat, the width survives slice / cast / IPC / Parquet round-trips, and any column-level operation that wants to know "how many bits does this hold" reads it off column.type.bits.

Why?

PyArrow's built-in uint8 / uint16 / uint32 / uint64 cover only the four power-of-two widths native to most CPUs. Protocol and binary formats routinely use other widths (10, 12, 14, 24, 48), and the usual workarounds — over-allocating (uint16 for a 12-bit field) or passing the width out-of-band in schema metadata — either lose the constraint on bitwise ops or drop it on the next slice. fletchr.uintn(bits=N) puts the width in the type and ships bit-width-safe kernels that keep padding bits zero across every operation.

Features

• Storage in the smallest native uint8 / uint16 / uint32 / uint64 container that fits N; padding bits above N are kept zero across construction and every operation.
• Lossless round-trip through Arrow IPC, Arrow Flight, and Parquet. Readers that don't have the extension registered see the raw uintN storage transparently — no exotic types in the wire format.
• Full Arrow null support via the standard validity bitmap.
• Bit-width-safe bitwise operators (~, &, |, ^, shifts) plus a fletchr_uintn.compute module (comparisons, set lookup, dedup / histogram / sort, popcount, bit-reversal, parity, exact reductions, conditional select, arithmetic with the overflow policy in the name, ...) — padding bits never leak.
• Cross-language wire format pinned in SPEC.md so Arrow readers in Java, C++, Go, R, JavaScript, etc. can implement compatible deserializers.

Install

uv add fletchr-uintn        # or: pip install fletchr-uintn

Requires Python 3.9+, NumPy 2.0+, and PyArrow 17+.

Quickstart

import pyarrow as pa
import pyarrow.parquet as pq
from fletchr_uintn import uintn_array

# 12-bit values — fit in a uint16 container, but the type knows it's 12 bits.
a = uintn_array([0, 1, 4095, None, 100], bits=12)
a.type            # UIntNType(bits=12)
a.to_pylist()     # [0, 1, 4095, None, 100]

# Bitwise ops respect the declared width: ~0 is 4095, not 65535.
(~a).to_pylist()  # [4095, 4094, 0, None, 3995]

# Composes as a column inside any pa.Table; round-trips through Parquet.
pq.write_table(pa.table({"x": a}), "out.parquet")
back = pq.read_table("out.parquet").column("x")
assert back.type.bits == 12

# Mismatched bit widths fail at the Arrow type system, not silently:
pa.concat_arrays([a, uintn_array([0, 1], bits=10)])  # raises ArrowInvalid

Public API

from fletchr_uintn import (
    UIntNType,             # the pa.ExtensionType
    UIntNArray,            # the pa.ExtensionArray (data structure + operator sugar)
    compute,               # named kernels: comparisons, set lookup, bitwise, reductions
    uintn_array,           # validated factory; dispatches on input type
    pack_bits,             # inverse of UIntNArray.unpack_bits
    to_numpy_zero_filled,  # numeric pa.Array -> ndarray, nulls read as 0
)

The extension type registers itself on import, so any pa.Table deserialized after import fletchr_uintn will surface UIntNArray columns instead of raw uintN storage.

fletchr_uintn.compute is to UIntNArray what pyarrow.compute is to pyarrow.Array: the array itself is a data structure (construction, indexing, conversions, whole-array ==, and & | ^ ~ << >> operator sugar), while every named element-wise / reduction operation is a free function that also accepts ChunkedArray columns:

import fletchr_uintn.compute as uc

uc.equal(a, 5)            # pa.BooleanArray mask (pc.equal lacks a kernel)
uc.is_in(a, [3, 5, 900])  # membership mask; unrepresentable values match nothing
uc.unique(a)              # distinct values, still uintn-typed
uc.bit_wise_reverse(a)    # width-aware bit reversal
uc.sum(a)                 # exact at every width — no uint64 wraparound
uc.if_else(mask, a, 0)    # conditional select, staying in the uintn domain
uc.subtract_wrap(a, b)    # mod-2**bits arithmetic (N-bit register semantics);
uc.add_checked(a, 1)      # ...or raise on overflow — no plain `add` by design
uc.modulo(a, 4)           # x % m (pyarrow has no modulo kernel)

(arr == 5 raises with guidance rather than silently comparing identity.) For any pyarrow.compute kernel compute doesn't wrap, arr.storage is the escape hatch: it's the raw uintN array, and every value is already masked to the declared width, so pc.<kernel>(arr.storage, ...) is safe — rewrap a result with uintn_array(result, bits=arr.bits) if you need the type back.

Links

• Source: https://github.com/fletchr-labs/fletchr
• Issues: https://github.com/fletchr-labs/fletchr/issues
• Wire format spec: SPEC.md

License

MIT.