NumPy Utils

NumPy structured array utilities — dtype construction, field views, joining, enum mapping, position arrays, byte/string conversion for C++ interop, and growable (std::vector-like) arrays.

Overview

vcti-nputils collects the low-level NumPy helpers shared across the vcti stack. Most of it is stateless functions over structured arrays and dtypes — building and reshaping dtypes, taking zero-copy field views, joining arrays, mapping enum values, and converting byte fields for pybind11/C++ interop. Alongside those it provides two small stateful containers for building a numpy array of unknown final size: GrowableArray (append-only — the common case) and DynamicArray (the same growth model plus soft deletion). Both grow on demand and hand off a plain contiguous array via to_numpy() for the read-heavy phase.

Which to use: append-only? GrowableArray. Need to remove elements mid-build? DynamicArray.

Installation

pip install vcti-nputils>=1.6.0

In pyproject.toml dependencies

dependencies = [
    "vcti-nputils>=1.6.0",
]

Quick Start

import numpy as np
from vcti.nputils import (
    as_ndarray,
    check_overflow,
    decode_field,
    drop_fields,
    DynamicArray,
    encode_field,
    fields_view,
    flatten_dtype,
    GrowableArray,
    join_struct_arrays,
    merge_adjacent_fields,
    name_array,
    position_array,
    rename_fields,
    structured_dtype,
    with_encoding,
)

# Join structured arrays horizontally
dt1 = np.dtype([('id', 'i4'), ('value', 'f8')])
dt2 = np.dtype([('name', 'U10')])
arr1 = np.array([(1, 1.5), (2, 2.5)], dtype=dt1)
arr2 = np.array([('Alice',), ('Bob',)], dtype=dt2)
joined = join_struct_arrays([arr1, arr2])
# dtype: [('id', 'i4'), ('value', 'f8'), ('name', 'U10')]

# Create a zero-copy view with selected fields
view = fields_view(joined, ['id', 'name'])

# Drop fields from a structured array (zero-copy)
clean = drop_fields(joined, ['value'])

# Build a structured dtype from a scalar dtype + names
coord_dt = structured_dtype('f8', ['x', 'y', 'z'])
# dtype([('x', '<f8'), ('y', '<f8'), ('z', '<f8')])

# Rename fields in a dtype
new_dt = rename_fields(dt1, {'id': 'node_id', 'value': 'temperature'})

# Flatten array fields into individual columns (default naming)
dt = np.dtype([('id', 'i4'), ('coords', 'f8', (3,))])
_, cols = flatten_dtype(dt)
# cols: ['id', 'coord_0', 'coord_1', 'coord_2']

# Flatten with explicit per-field names
_, cols = flatten_dtype(dt, field_names={'coords': ['x', 'y', 'z']})
# cols: ['id', 'x', 'y', 'z']

# Flatten with a custom format string
_, cols = flatten_dtype(dt, fmt="{name}[{dim}]")
# cols: ['id', 'coord[0]', 'coord[1]', 'coord[2]']

# Merge adjacent 'S' fields into one (pure dtype view). Multiple merges
# can be specified at once; same-field overlap and name collisions are
# validated before anything is returned.
dt = np.dtype([
    ('first', 'S4'), ('last', 'S6'),
    ('city', 'S8'), ('state', 'S2'),
    ('age', 'i4'),
])
merged = merge_adjacent_fields(dt, {
    'name':    ['first', 'last'],
    'address': ['city', 'state'],
})
# dtype([('name', 'S10'), ('address', 'S10'), ('age', '<i4')])

# Map numeric enum values to names
enum_dict = {1: 'ACTIVE', 2: 'INACTIVE', 3: 'PENDING'}
names = name_array(np.array([1, 2, 1, 3]), enum_dict)

# Convert counts to cumulative offsets
offsets = position_array(np.array([3, 2, 4, 1]))
# array([0, 3, 5, 9, 10])

# Safely coerce inputs to ndarray
arr = as_ndarray([1, 2, 3], dtype=np.float64)
empty = as_ndarray(None)  # array([], dtype=float64)

# Byte <-> string conversion for C++/pybind11 interop
dt = np.dtype([('name', 'S10'), ('name_length', 'i4')])
sa = np.zeros(2, dtype=dt)
encode_field(sa, 'name', ['Alice', 'Bob'], length_field='name_length')
decoded = decode_field(sa, 'name')
overflow = check_overflow(sa, 'name', 'name_length')

# Attach encoding to a dtype so decode_field/encode_field use it automatically
name_dt = with_encoding(np.dtype('S32'), 'latin-1')

# Build an array incrementally without knowing the final size (append-only)
ga = GrowableArray(np.dtype([('id', 'i4'), ('value', 'f8')]))
ga.append((1, 1.5))
ga.extend([(2, 2.5), (3, 3.5)])
result = ga.to_numpy()  # independent, exact-size array you own

# Need to remove elements mid-build? DynamicArray adds soft deletion
da = DynamicArray(np.dtype('i8'))
da.extend([10, 20, 30, 40])
da.delete(1)            # elements shift: da[1] is now 30
clean = da.to_numpy()   # array([10, 30, 40])

These are typed, numpy-backed accumulators, not faster lists. For a method-by-method comparison, measured trade-offs, and guidance on when to use them versus a Python list, see docs/design/comparison.md; for usage recipes, docs/patterns.md. Reproduce the numbers with python benchmarks/benchmark_growable_array.py.

Module layout

Each category lives in its own module. All public functions are re-exported from vcti.nputils.

Functions

Dtype construction & transformation

Zero-copy views

Joining

Mapping, offsets, coercion

Containers

Byte / string conversion (pybind11 interop)

Dependencies

• numpy (>=2.0)