fastdb4py 0.1.13

FastCarto database bindings

fastdb

A C++ local database library with Python bindings (via SWIG). Designed as a fast, lightweight, and easy-to-use data communication layer for RPC and coupled modeling in scientific computing.

Core design goals:

• Zero-copy columnar access — field data is exposed directly as NumPy arrays backed by C++ memory, no serialization overhead
• Ref-graph support — Features can reference other Features across tables, forming typed object graphs
• Shared memory IPC — publish a database to POSIX/Windows shared memory and read it from other processes with zero-copy
• File persistence — save and load databases as compact binary files

What's new

• 2026-03-17 (Release 0.1.13): Fixed two C++ correctness bugs in the batch field read/write API: (1) getFieldsAsDoubles now correctly handles U8/U16/U32/I32 fields (previously returned NAN); (2) set_field_value_t now correctly writes U16N normalized fields (missing memcpy caused silent data loss). Also added batch scalar field API (read_all_scalars / write_all_scalars) with up to 12× speedup over per-field access.
• 2026-03-04 (Release 0.1.12): Fixed a critical issue where loading large database files (> 2GB) on Linux/Unix systems would fail to read the complete file, leading to missing tables or data corruption. The file reading logic has been improved to correctly handle partial reads for large files. (PR #23)
• 2026-03-04 (Memory Overflow Improvement): Enhanced the MemoryStream implementation to handle large data sizes exceeding 4GB without causing size overflow in chunk_data_t.size (u32). This improvement allows for more robust handling of large datasets in memory. (PR #22)
• 2026-02-28 (Release Improvement): Fix bugs related to build process in Windows. (PR #20)
• 2025-12-31 (Bug Fix): Fixed an issue where shared memory segments were not being properly unregistered from the resource tracker upon closing, which could lead to resource leaks. (PR #17)
• 2025-12-15 (Release Improvement): Enabled distribution of pre-compiled binary wheels for macOS (Intel/Apple Silicon) and Linux (x86_64/aarch64), eliminating the need for local compilation tools during installation. (PR #15)
• 2025-12-10 (Bug Fix): Fixed the data type mapping for U32 fields in Python bindings to ensure correct representation as unsigned 32-bit integers in NumPy arrays. (PR #13)
• 2025-12-10 (Bug Fix): Fixed an out-of-bounds access issue in FastVectorDbLayer::Impl::getFieldOffset() when the field index is equal to the field count. (PR #12)
• 2025-12-10 (Performance Improvement): Modified ORM.truncate() to support directly allocating features without initializing them for performance consideration. Note that this change may have side effects; please test thoroughly. (PR #11)

Installation

pip install fastdb4py

Pre-compiled binary wheels are provided for major platforms (macOS Intel/Apple Silicon, Linux x86_64/aarch64, Windows AMD64). For other systems, the package builds from source and requires a C++ compiler, CMake, and SWIG.

Quick Start

import fastdb4py as fx
import numpy as np

# 1. Define schema
class Point(fx.Feature):
    x: fx.F64
    y: fx.F64
    z: fx.F64

# 2. Create a fixed-size table and fill via NumPy (fastest path)
N = 10_000
db = fx.ORM.truncate([fx.TableDefn(Point, N)])
tbl = db[Point][Point]

tbl.column.x[:] = np.linspace(0, 1, N)
tbl.column.y[:] = np.zeros(N)
tbl.column.z[:] = np.ones(N)

# 3. Read back as NumPy arrays (zero-copy)
xs = tbl.column.x   # numpy array backed by C++ memory
print(xs[:5])

# 4. Save and reload
db.save("my_data")
db2 = fx.ORM.load("my_data", from_file=True)

Field Types

Python type	C++ storage	Description
fx.U8	uint8_t	Unsigned 8-bit integer
fx.U16	uint16_t	Unsigned 16-bit integer
fx.U32	uint32_t	Unsigned 32-bit integer
fx.I32	int32_t	Signed 32-bit integer
fx.F32	float	32-bit float
fx.F64	double	64-bit float
fx.STR	string table index	Short ASCII string
fx.WSTR	string table index	Unicode string
fx.BYTES	blob	Raw bytes / geometry chunk
fx.U8N	uint8_t	Normalized float in [vmin, vmax], stored as u8
fx.U16N	uint16_t	Normalized float in [vmin, vmax], stored as u16
OtherFeature	ref	Cross-table reference to another Feature

Usage

Creating a Database

Fixed-size tables (ORM.truncate)

Use ORM.truncate when you know the number of rows up front. This pre-allocates the full table in one shot and enables the fastest columnar write path.

import fastdb4py as fx
import numpy as np

class Particle(fx.Feature):
    x: fx.F64
    y: fx.F64
    vx: fx.F64
    vy: fx.F64
    mass: fx.F32

N = 100_000
db = fx.ORM.truncate([fx.TableDefn(Particle, N)])
tbl = db[Particle][Particle]

# Vectorized write — 1 memcpy per column, ~1000× faster than row-wise push
tbl.column.x[:]    = np.random.uniform(-1, 1, N)
tbl.column.y[:]    = np.random.uniform(-1, 1, N)
tbl.column.vx[:]   = np.zeros(N)
tbl.column.vy[:]   = np.zeros(N)
tbl.column.mass[:] = np.ones(N, dtype=np.float32)

Multiple tables with different schemas can be created in one call:

class Cell(fx.Feature):
    id: fx.U32
    temperature: fx.F64

db = fx.ORM.truncate([
    fx.TableDefn(Particle, 50_000),
    fx.TableDefn(Cell, 1_000),
])

Dynamic tables (ORM.create + push)

Use ORM.create when the number of rows is not known in advance.

import fastdb4py as fx

class LogEntry(fx.Feature):
    level: fx.U8
    code: fx.U32
    message: fx.STR

db = fx.ORM.create()

db.push(LogEntry(level=1, code=200, message="ok"))
db.push(LogEntry(level=3, code=500, message="internal error"))

db._combine()   # finalize: serialize + reload as immutable db

Reading Data

Columnar access (zero-copy NumPy)

The fastest read path. table.column.field_name returns a NumPy array that is a direct view of the underlying C++ memory — no copy, no allocation.

db = fx.ORM.load("my_data", from_file=True)
tbl = db[Particle][Particle]

xs = tbl.column.x    # np.ndarray, dtype=float64, shape=(N,)
ys = tbl.column.y

# Vectorized operations in-place
xs += 0.01 * tbl.column.vx
ys += 0.01 * tbl.column.vy

# NumPy aggregations
print(f"Mean x: {xs.mean():.4f}")
print(f"Max speed: {np.sqrt(tbl.column.vx**2 + tbl.column.vy**2).max():.4f}")

Row-wise iteration

# Standard iterator — allocates a new Feature wrapper per row
for p in tbl:
    print(f"x={p.x:.3f}  y={p.y:.3f}")

# High-performance iterator — reuses the same wrapper (no per-row allocation)
# Do NOT hold references to the yielded object across loop iterations.
for p in tbl.iter_reuse():
    print(f"x={p.x:.3f}  y={p.y:.3f}")

Index access

first = tbl[0]
print(f"Particle 0: x={first.x}, mass={first.mass}")

Feature References (Object Graphs)

Features can store typed references to features in other tables. Assign a Feature instance to a ref field; cross-table links are resolved automatically on read.

import fastdb4py as fx

class Point(fx.Feature):
    x: fx.F64
    y: fx.F64
    z: fx.F64

class Triangle(fx.Feature):
    a: Point    # ref field — links to a Point feature
    b: Point
    c: Point

db = fx.ORM.truncate([
    fx.TableDefn(Point, 6),
    fx.TableDefn(Triangle, 2, 'TriA'),
    fx.TableDefn(Triangle, 2, 'TriB'),
])

points = db[Point][Point]
for i in range(6):
    points[i].x = float(i)
    points[i].y = float(i) * 0.5
    points[i].z = 0.0

tri = db[Triangle]['TriA'][0]
tri.a = points[0]
tri.b = points[1]
tri.c = points[2]

# Read back — ref is resolved transparently
print(tri.a.x, tri.b.x, tri.c.x)   # 0.0  1.0  2.0

File Persistence

# Save to disk
db.save("simulation_state")

# Load from disk
db = fx.ORM.load("simulation_state", from_file=True)
tbl = db[Particle][Particle]

# Access, modify, then save again
tbl.column.x[:] += 1.0
db.save("simulation_state")

Shared Memory IPC

Share a database across processes with zero-copy. The publisher writes once; all readers map the same physical memory.

# --- Process A: publisher ---
import fastdb4py as fx

class Signal(fx.Feature):
    t: fx.F64
    value: fx.F64

db = fx.ORM.create()
db.push(Signal(t=0.0, value=3.14))
db.push(Signal(t=0.1, value=2.71))

db.share("my_signals", close_after=True)  # publish and release local handle

# --- Process B: reader (runs concurrently) ---
import fastdb4py as fx

class Signal(fx.Feature):
    t: fx.F64
    value: fx.F64

db = fx.ORM.load("my_signals")   # load from shared memory
tbl = db[Signal][Signal]

for s in tbl:
    print(f"t={s.t}  value={s.value}")

db.unlink()   # release shared memory segment when done

Batch Scalar Access

For db-mapped Features (e.g. from table[i] or table.iter_reuse()), you can read or write all scalar fields in a single C++ call instead of one SWIG call per field. This is especially useful in tight loops or when building custom row-processing kernels.

import numpy as np

class Point(fx.Feature):
    x: fx.F64
    y: fx.F64
    z: fx.F64

# ... (db setup as above) ...
feat = tbl[0]

# Read all scalar fields into a float64 array (1 SWIG call)
out = np.empty(3, dtype=np.float64)
feat.read_all_scalars(out)   # fills out in-place; returns out
# out[0] = x, out[1] = y, out[2] = z

# Write all scalar fields from a float64 array (1 SWIG call)
feat.write_all_scalars(np.array([1.0, 2.0, 3.0]))

Field order in the array follows the order they are declared in the Feature class.

Performance Notes

Pattern	Throughput	Notes
table.column.x[:] columnar read/write	~100 ns for any N	Zero-copy NumPy view, 1 SWIG call
Table.fill(field, array)	~2 µs per column	1 SWIG call + memcpy
feature.read_all_scalars()	~200 ns for 3 fields	1 SWIG call for all scalar fields
table.iter_reuse() row access	~350 ns/row	Reuses Feature wrapper, no allocation
for feat in table row access	~1.2 µs/row	Allocates Feature wrapper per row
feat.x single field read (db-mapped)	~420 ns	1 SWIG call

Recommended patterns by use case:

• Bulk read/write of one field across all rows → table.column.x (columnar, zero-copy)
• Bulk fill all fields from arrays → ORM.truncate + table.column.field[:] = array
• Iterate and process all fields per row → table.iter_reuse() + feat.read_all_scalars()
• Sparse random access → table[i].field

Development

This project uses DevContainer for the development environment. See .devcontainer/devcontainer.example.json for configuration details. Requires Docker/Podman and the VSCode DevContainer extension.

The py_utils.sh script handles common development tasks:

./py_utils.sh --clean   # remove C++ build artifacts and SWIG-generated bindings
./py_utils.sh --build   # build C++ core + Python bindings
./py_utils.sh --test    # run Python unit tests

Build requirements: C++17 compiler, CMake >= 3.16, SWIG >= 4.0, NumPy.