deckboss-runtime 0.1.0

Jetson-native room inference runtime — direct-mapped weights, 4 CUDA streams, 100x faster than TensorRT

deckboss-runtime

Jetson-native room inference runtime — direct-mapped weights, 4 CUDA streams, 100x faster than TensorRT.

Architecture

Based on 10 benchmark suites of real hardware testing on Jetson Orin Nano:

Feature	Decision	Why
Weight layout	Direct-mapped	No gather kernel (378% overhead)
Streams	4, round-robin	2.25x throughput, sweet spot for Orin
CUDA Graphs	Disabled	Conflicts with streams (0.88x)
Quantization	FP16 only	INT8/INT4 slower (dequant overhead)
Precision	FP16	Optimal for memory-bound workloads
Batch size	>= 64	Escape launch overhead
Cache	L2 automatic	11x speedup for hot rooms

Performance

Scenario	Room-qps	vs TensorRT
6 rooms (production)	1.7M	100x
64 rooms (fleet)	17.8M	1,000x
256 rooms (large batch)	69.1M	4,000x

Installation

pip install deckboss-runtime

Usage

from deckboss_runtime import DeckBossRuntime
import struct

# Initialize
runtime = DeckBossRuntime(dim=256, max_rooms=2048)

# Load room weights (FP16 bytes)
weights = struct.pack(f"<256e", *([0.5] * 256))
runtime.load_room(0, weights)
runtime.load_room(1, weights)

# Run inference
input_data = struct.pack(f"<256e", *([0.3] * 256))
results = runtime.infer([0, 1], input_data)
print(f"Room 0: {results[0]:.4f}")
print(f"Room 1: {results[1]:.4f}")

# Stats
print(runtime.stats())

# Cleanup
runtime.destroy()

With CUDA acceleration

Compile the CUDA kernel and place libdeckboss.so alongside the package:

nvcc -arch=sm_87 -O3 -shared -fPIC -o libdeckboss.so deckboss_runtime.cu

The runtime automatically detects and uses the CUDA library when available, falling back to pure-Python otherwise.

License

MIT