Hypertile

A unified, work-stealing executor collocating free-threaded Python coroutines and Send Rust futures in one right-sized thread pool.

1. Executive Summary & Problem Space

Modern high-performance applications combining Python and Rust (such as FastAPI web services, AI/ML inference servers, and distributed ETL pipelines) conventionally run two completely independent async runtime stacks:

1. Python's Async Stack: Single-threaded asyncio event loop driving coroutines, combined with a separate ThreadPoolExecutor (allocating 8–32 OS threads) for offloading blocking work.
2. Rust's Async Stack: A native multi-threaded runtime (Tokio-style) with its own thread pool sized to available CPU cores.

The Hidden Bottlenecks of Dual Pools:

• 💥 Hardware Oversubscription: Two independent pools sized against hardware cores create $2 \times \text{CPUs}$ active OS threads, resulting in relentless thread parking, context switching, and cache-line invalidation.
• 🐢 Double-Hop Boundary Latency: Crossing between Python and Rust requires two scheduling hops: $$\text{Native Wake} \longrightarrow \text{Native Tick} \longrightarrow \texttt{call_soon_threadsafe} \longrightarrow \texttt{eventfd} \longrightarrow \text{Asyncio Tick} \longrightarrow \text{Python Task}$$
• 🔒 Zero Capacity Sharing: Idle interpreter threads cannot assist with stranded native CPU work, and idle native threads cannot step Python coroutines.
• 🛑 GIL Overhead vs. Free-Threaded Promise: Standard CPython serializes bytecode execution under the Global Interpreter Lock (GIL). However, with PEP 779 free-threaded Python (3.13t / 3.14t+), Python bytecode can execute truly in parallel across multiple OS threads—if and only if the executor is designed to step coroutines natively without lock contention.

2. Architecture & Core Innovations

Hypertile provides one unified multi-threaded runtime whose workers are individually either native-only (pure Rust) or bilingual (interpreter-attached):

                     +-----------------------------------------------+
                     |            HYPERTILE GLOBAL POOL              |
                     |   shared injector (MPSC) + per-worker local   |
                     |   Chase-Lev deques (crossbeam-deque)          |
                     +-----------------------------------------------+
                           |                   |                  |
             +-------------+---------+  +------+------+   +------+---------+
             |  Bilingual worker    |  |  Native     |   |  Bilingual     |
             |  (interpreter-attach |  |  worker     |   |  worker        |
             |   on 3.13t/3.14t+)   |  |  (Rust-only)|   |  (… N workers) |
             +----------------------+  +-------------+   +----------------+
                  |          |                |
        +---------+          |                +------------------+
        | Step Python        |                |  Poll Rust       |
        | coroutines, in     |                |  futures (any    |
        | batches, via       |                |  Send future,    |
        | coro.send()        |                |  never two       |
        | (interpreter       |                |  threads at once)|
        |  mode)             |                |                  |
        +--------------------+                +------------------+

Architectural Highlights:

1. Single-Hop Continuation Handoff: When a native task finishes on worker $W$, $W$ pushes the awaiting continuation directly onto its own local Chase-Lev deque. $W$ resumes it immediately without returning to an event loop or waking another thread (~1.01 µs per hop).
2. Cache-Line False-Sharing Elimination (CachePadded): Hot atomics (idle_count) and worker locks (idle_stack, stealers_cache) are isolated via crossbeam_utils::CachePadded, eliminating MESI/MOESI cache-line bouncing across 64-byte (x86/ARM) and 128-byte (Apple Silicon M-series/POWER) lines.
3. Sub-Nanosecond PRNG (fast_rand): Replaced thread-local cryptographic RNG calls with an ultra-fast non-cryptographic SplitMix64 PRNG, executing victim selection in 2–3 single-cycle ALU instructions.
4. Inter-Core Batch Work-Stealing (steal_batch_and_pop): When stealing from the global injector or peer workers, an idle thread atomically claims half of the victim's queue in a single transaction, executing remaining tasks locally out of L1 cache with zero contention.
5. Hardware-Adaptive Vectorized Batching: Vectorized APIs (batch_native_pipeline and gather_to_thread) cross the Python $\leftrightarrow$ Rust FFI boundary once per slice and dynamically chunk workloads across physical cores, yielding >237,000 to >697,000 operations/sec.
6. Dynamic Worker Registration & Capacity Sharing: External server threads (e.g. FastAPI / Uvicorn workers) dynamically join Hypertile's work-stealing pool as auxiliary workers while idle and leave cleanly in 2.76 µs / cycle.
7. Panic Containment: Native panics are caught via catch_unwind and surfaced as hypertile.PanicInTask without poisoning mutexes or killing worker threads.

3. GIL vs. Free-Threaded Support (Honest Matrix)

4. API Reference & Developer Ergonomics

Asynchronous Offloading: hypertile.to_thread

Ultra-low-latency drop-in replacement for asyncio.to_thread(). Dispatches callables directly to Hypertile's bilingual workers without allocating OS threads or intermediate asyncio futures:

import hypertile

# Dispatch arbitrary Python callable
result = await hypertile.to_thread(crypto_hash, payload, rounds=50)

Vectorized Parallel Dispatch: hypertile.gather_to_thread

Vectorized parallel execution across an iterable of inputs. Crosses the FFI boundary once for the entire batch:

# Process a collection of items in parallel across bilingual workers
results = await hypertile.gather_to_thread(process_record, [r1, r2, r3, r4])

Function Decorator: @hypertile.task

Converts synchronous functions into awaitable Hypertile tasks:

@hypertile.task
def verify_token(raw_jwt: str) -> dict:
    return jwt.decode(raw_jwt, key, algorithms=["RS256"])


# Inside an async endpoint:
claims = await verify_token(header_auth)

Native Pipelines: hypertile.spawn_native_pipeline

Route CPU-intensive numerical and cryptographic transforms directly onto the native Rust work-stealing queue with single-hop continuation:

native_task = hypertile.spawn_native_pipeline(raw_bytes, rounds=100)
digest = await native_task

High-Throughput Batch Pipeline: hypertile.batch_native_pipeline

Submits a batch of payloads directly into the native work-stealing engine in a single FFI crossing:

# Dispatches thousands of payloads with sub-5µs amortized latency
digests = await hypertile.batch_native_pipeline(payload_chunks, rounds=100)

Dynamic Worker Registration

Join the work-stealing pool from long-running server threads (e.g., FastAPI lifespan):

with hypertile.register_worker(kind="bilingual") as worker:
    # Assist the executor while waiting for requests
    worker.run_until_idle()

Cooperative Cancellation: CancellationToken

Propagate cooperative cancellation across bilingual and native workers:

token = hypertile.CancellationToken()
token.cancel()
assert token.is_cancelled()

5. C / C++ Embeddable ABI (hypertile-capi & include/hypertile.h)

For applications written in C, C++, Go (cgo), or Zig, Hypertile provides a lightweight, zero-overhead extern "C" ABI via the hypertile-capi crate and include/hypertile.h.

No heavy C++ framework is imposed; the C ABI maps directly to Hypertile's Chase-Lev deques and work-stealing pool without intermediate runtime overhead.

Building the C ABI Libraries

# Build release dynamic (.dll/.so/.dylib) and static (.lib/.a) libraries
cargo build --release -p hypertile-capi

Artifacts are generated in target/release/:

• Dynamic Library: hypertile_capi.dll (Windows) / libhypertile_capi.so (Linux) / libhypertile_capi.dylib (macOS)
• Static Library: hypertile_capi.lib (MSVC) / libhypertile_capi.a (GCC/Clang)

Core C API Functions:

• hypertile_init(size_t num_workers): Initialize global pool (pass 0 for CPU core count).
• hypertile_spawn(work, arg): Asynchronously dispatch a task; returns an opaque hypertile_task_t*.
• hypertile_wait(task, &out_result): Efficiently park the calling thread until task completes.
• hypertile_poll(task, &out_result): Non-blocking completion poll (0 ready, 1 pending).
• hypertile_task_destroy(task): Free task handle (safe before or after completion).
• hypertile_spawn_with_callback(work, arg, callback, user_data): Fire-and-forget task with completion callback.
• hypertile_batch_spawn(work, args, out_results, count): Vectorized parallel batch execution (>5,000,000 items/sec).
• hypertile_register_worker(): Dynamically join the work-stealing pool from external C/C++ threads.
• hypertile_shutdown(): Cleanly drain and stop worker threads.

C Example

#include <stdio.h>
#include <stdint.h>
#include "hypertile.h"

void* compute(void* arg) {
    uintptr_t x = (uintptr_t)arg;
    return (void*)(x * 2);
}

int main(void) {
    hypertile_init(0); // auto CPU cores

    // 1. Single Task Spawning
    hypertile_task_t* task = hypertile_spawn(compute, (void*)21);
    void* result = NULL;
    hypertile_wait(task, &result);
    printf("Result: %zu\n", (uintptr_t)result); // 42
    hypertile_task_destroy(task);

    // 2. Vectorized Parallel Batch (10,000 items)
    const size_t COUNT = 10000;
    void* args[COUNT];
    void* results[COUNT];
    for (size_t i = 0; i < COUNT; ++i) args[i] = (void*)(uintptr_t)i;
    hypertile_batch_spawn(compute, args, results, COUNT);

    hypertile_shutdown();
    return 0;
}

Compiling and Linking:

# GCC / Clang (Dynamic link)
gcc -O3 -I include main.c -L target/release -lhypertile_capi -o app

# MSVC (cl.exe)
cl /O2 /I include main.c target\release\hypertile_capi.dll.lib

A complete executable verification program is located at examples/c/main.c.

6. Benchmarks & Empirical Performance

All benchmarks were measured on Windows AMD64 (8 physical cores / 16 threads) comparing identical cryptographic/numerical workloads:

A. Free-Threaded (No-GIL, Python 3.13t) Benchmark:

.venv-313t/Scripts/python showcase/free_threaded_showcase.py

B. Standard GIL (Python 3.11) Honest Benchmark:

.venv/Scripts/python showcase/showcase_benchmark.py

7. Development with uv & Multi-Environment Setup

Hypertile strictly recommends Astral uv for fast, reproducible virtual environment management:

1. Free-Threaded Environment (Python 3.13t)

# Install free-threaded Python 3.13t
uv python install 3.13t

# Create virtual environment
uv venv --python 3.13t .venv-313t

# Install dependencies with uv
uv pip install maturin pytest fastapi httpx --python .venv-313t/Scripts/python.exe

# Build and install editable release wheel
$env:VIRTUAL_ENV = "d:\HyperTile\.venv-313t"
& .venv-313t\Scripts\maturin.exe develop --release --uv

2. Standard GIL Environment (Python 3.11)

# Create virtual environment
uv venv --python 3.11 .venv

# Install dependencies with uv
uv pip install maturin pytest fastapi httpx --python .venv/Scripts/python.exe

# Build and install editable release wheel
$env:VIRTUAL_ENV = "d:\HyperTile\.venv"
& .venv\Scripts\maturin.exe develop --release --uv

8. Production Examples

Complete, executable production examples are provided in examples/:

1. C / C++ Embeddable Driver (examples/c/main.c): Zero-overhead C API driver testing single tasks, polling, callbacks, vectorized batch execution, and worker registration.
2. FastAPI Microservice (examples/fastapi_service.py): Lifespan worker registration, @hypertile.task offloading, and native pipeline endpoints.

   python examples/fastapi_service.py
   

3. Batch Data Pipeline (examples/data_pipeline.py): Multi-stage ETL pipeline, cooperative cancellation tokens, dynamic worker scaling, and vectorized native batches.

   python examples/data_pipeline.py
   

9. Multi-OS & Hardware Compatibility Matrix

10. License

Dual-licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)