idempotent-rl: Zero-Copy In-Place Multi-Tensor Replay Buffer Compactor for Deep RL

Eliminate 100% of auxiliary VRAM allocations during Prioritized Experience Replay (PER) multi-tensor compaction in CleanRL, Stable-Baselines3, Isaac Gym, and TorchRL.

🚀 The Bottleneck: Multi-Tensor Buffer Reallocation

In deep reinforcement learning (RL) and continuous robotics control (e.g. Isaac Gym, Brax, CleanRL), vectorized replay buffers store heterogeneous transition tuples: $$\mathcal{T} = (\mathbf{s}, \mathbf{a}, r, \mathbf{s}', d)$$

When prioritizing experiences via Prioritized Experience Replay (PER), top-priority transitions are periodically consolidated or evicted. Standard deep learning runtimes execute this compaction via out-of-place multi-tensor gathering (torch.gather / dynamic cudaMalloc):

1. Multiplied Auxiliary VRAM Spikes: Because states, actions, rewards, next-states, and done flags must each be duplicated in destination memory, auxiliary VRAM overhead scales multiplicatively ($O(N \cdot (2 D_s + D_a + 2))$), consuming hundreds of megabytes to gigabytes of transient accelerator memory.
2. Double-Buffering Bus Congestion: Copying entire transition sets across global memory saturates GPU high-bandwidth memory (HBM).
3. Allocation Jitter in Real-Time Simulators: Dynamic memory allocation stalls simulation steps during high-throughput vectorized actor training.

⚡ The Solution: In-Situ Multi-Tensor Idempotent Permutation

idempotent-rl reorganizes all five heterogeneous transition tensors simultaneously in-place directly within existing memory allocations using $O(1)$ scalar hardware registers:

• Idempotent Attractor Basins: Enforces the algebraic property $f(f(x)) = f(x)$, locking high-priority transitions into stabilized contiguous memory $[0, K-1]$.
• Simultaneous Multi-Tensor Permutation: Traverses disjoint cycle structures on GPU thread blocks, swapping $(s, a, r, s', d)$ synchronously in register space without auxiliary global buffers.
• In-Register 2-Cycle Fast-Path: Mutually transposed transitions are swapped directly across thread registers with zero memory overhead.
• 100% Zero Auxiliary VRAM: Exactly 0.00 MB auxiliary secondary memory allocated.
• Bit-Exact Numerical Parity: Zero approximation error ($\Delta = 0.000000$, 0 NaN).
• Blazing Throughput: Exceeds 68+ Million transitions/second on modern NVIDIA GPUs.

📊 Benchmark: NVIDIA RTX PRO 500 Blackwell (sm_120)

Workload: 32 Environments, 8,192 Transitions/Env (262,144 transitions total), StateDim=64, ActionDim=16, Capacity=4,096 (50% Compaction, float32)

📦 Installation

git clone https://github.com/aemre-cetin/idempotent-rl.git
cd idempotent-rl
pip install -e .

Requirements: torch >= 2.0.0, triton >= 2.1.0.

🛠️ Quickstart

import torch
from idempotent_rl import InplacePERCompactor

# Initialize compactor with state and action feature dimensions
compactor = InplacePERCompactor(state_dim=64, action_dim=16)

# Vectorized replay buffer transition tensors on GPU [Envs, Transitions, ...]
states = torch.randn((32, 8192, 64), dtype=torch.float32, device="cuda")
actions = torch.randn((32, 8192, 16), dtype=torch.float32, device="cuda")
rewards = torch.randn((32, 8192), dtype=torch.float32, device="cuda")
next_states = torch.randn((32, 8192, 64), dtype=torch.float32, device="cuda")
dones = torch.zeros((32, 8192), dtype=torch.float32, device="cuda")

# TD-errors computed during Q-learning / actor-critic updates
td_errors = torch.rand((32, 8192), dtype=torch.float32, device="cuda")

# In-place consolidation: 0 bytes auxiliary global memory allocated
compacted_s, compacted_a, compacted_r, compacted_ns, compacted_d = compactor.compact(
    states, actions, rewards, next_states, dones,
    td_errors=td_errors, capacity=4096
)

# Output tensors are physically contiguous: [32, 4096, ...]
print("Compacted states shape:", compacted_s.shape)

CleanRL Integration Hook

from idempotent_rl.integrations import CleanRLReplayBufferInplaceHook

# Drop-in hook for CleanRL vectorized environments
hook = CleanRLReplayBufferInplaceHook(state_dim=64, action_dim=16, capacity=4096)

# Execute in-place compaction during buffer maintenance
s, a, r, ns, d = hook(rb_states, rb_actions, rb_rewards, rb_next_states, rb_dones, td_errors)

🛡️ Patent & Intellectual Property Notice

The mathematical formulations, state-transition architectures, and in-situ hardware compaction kernels implemented in this library are protected under pending patent application with the United States Patent and Trademark Office:

• U.S. Patent Application Number: 64/148,668
• Confirmation Number: 5890
• Status: PATENT PENDING
• First Named Inventor: Dr. Ahmet Emre ÇETİN

Academic evaluation, non-commercial research, and open-source collaboration are permitted under the terms of the Apache 2.0 License. Commercial deployment in proprietary hardware or commercial cloud runtimes is subject to bilateral licensing agreements with the author.

📜 Academic Citation

@article{cetin2026idempotentrl,
  title={Zero-Copy In-Place Multi-Tensor Compaction and Idempotent Associative Permutations in Deep Reinforcement Learning Replay Buffers},
  author={Cetin, A. Emre},
  journal={arXiv preprint},
  year={2026},
  note={U.S. Patent Application No. 64/148,668}
}

@article{cetin2013idempotent,
  title={Idempotent Permutations},
  author={Cetin, A. E.},
  journal={arXiv:1307.3877 [cs.DS]},
  year={2013}
}

📄 License

Licensed under the Apache License, Version 2.0. Copyright © 2026 Dr. A. Emre ÇETİN. All Rights Reserved.