Turbo-Torch

Turbo-Torch is a performance-oriented drop-in interface inspired by PyTorch, designed around the idea of reducing repeated computation overhead through internal caching, Cython-backed execution paths, and lightweight runtime optimizations.

The goal is simple:

Keep the PyTorch-style developer experience, while making repeated workloads feel faster.

Turbo-Torch preserves familiar tensor operations, module APIs, optimizers, and utility patterns while introducing an internal optimization layer that can cache reusable intermediate state and avoid unnecessary Python-level overhead.

Installation

pip install turbo-torch

Quick Start

Turbo-Torch follows the familiar PyTorch programming model.

import turbo_torch as torch
from turbo_torch import nn
from turbo_torch import optim

This allows existing PyTorch-style code to remain largely unchanged while using Turbo-Torch's runtime layer.

Example

import turbo_torch as torch
from turbo_torch import nn

model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Linear(256, 10)
)

x = torch.randn(64, 784)

output = model(x)

print(output.shape)

Why Turbo-Torch?

Traditional tensor workloads can repeatedly perform the same bookkeeping and dispatch operations:

Python
  ↓
Tensor API
  ↓
Operator dispatch
  ↓
Kernel execution

Turbo-Torch introduces an internal optimization layer:

Python
  ↓
Turbo-Torch API
  ↓
Cython Runtime Layer
  ↓
Cache / Dispatch Layer
  ↓
Tensor Operations
  ↓
Backend Kernel

Frequently reused execution paths can therefore bypass portions of the normal Python-side dispatch overhead.

Internal Architecture

Turbo-Torch is built around several internal components.

Runtime Cache

The runtime maintains lightweight caches for reusable execution metadata, including:

• operator dispatch information
• tensor shape signatures
• dtype/device combinations
• frequently accessed execution paths
• reusable intermediate metadata
• module-level execution state

A simplified lookup can be thought of as:

(operation, shape, dtype, device)
              ↓
        cache lookup
        ↙         ↘
    HIT             MISS
     ↓                ↓
cached path       resolve path
     ↓                ↓
execution ←────── cache update

The cache is designed to reduce repeated resolution work rather than blindly caching tensor values.

Cython Execution Layer

Performance-sensitive runtime components are implemented through Cython-oriented paths where appropriate.

Instead of performing every piece of dispatch logic through Python objects, Turbo-Torch can move selected hot-path operations closer to the CPython C-API boundary.

Conceptually:

Python API
    ↓
Cython bridge
    ↓
typed runtime structures
    ↓
cached dispatch
    ↓
backend operation

This reduces Python interpreter overhead for workloads containing large numbers of small or repeatedly invoked operations.

Drop-In PyTorch Style

Turbo-Torch intentionally follows familiar PyTorch conventions.

For example:

import turbo_torch as torch
import turbo_torch.nn as nn
import turbo_torch.optim as optim

Common APIs retain their expected usage patterns:

x = torch.tensor([1, 2, 3])

model = nn.Linear(3, 2)

optimizer = optim.Adam(
    model.parameters(),
    lr=1e-3
)

The intention is that developers should not need to learn an entirely new tensor programming model just to take advantage of the runtime layer.

Tensor Operations

a = torch.randn(1024, 1024)
b = torch.randn(1024, 1024)

c = torch.matmul(a, b)

Repeated operations can benefit from cached runtime metadata:

for _ in range(1000):
    c = torch.matmul(a, b)

Turbo-Torch's optimization layer can reuse information associated with previously resolved execution paths where the workload characteristics remain compatible.

Neural Networks

Turbo-Torch supports the familiar module-oriented programming model:

class Network(nn.Module):
    def __init__(self):
        super().__init__()

        self.layers = nn.Sequential(
            nn.Linear(784, 512),
            nn.ReLU(),
            nn.Linear(512, 10)
        )

    def forward(self, x):
        return self.layers(x)

Training remains familiar:

model = Network()
optimizer = optim.Adam(model.parameters())

for x, y in dataloader:
    optimizer.zero_grad()

    output = model(x)
    loss = loss_fn(output, y)

    loss.backward()
    optimizer.step()

Optimization Strategy

Turbo-Torch focuses primarily on reducing runtime overhead around tensor execution, rather than attempting to replace the underlying numerical backend.

The optimization stack can be summarized as:

┌─────────────────────────────┐
│       Turbo-Torch API       │
├─────────────────────────────┤
│     Runtime Dispatch        │
├─────────────────────────────┤
│   Cython Optimization Layer │
├─────────────────────────────┤
│     Internal Cache Layer     │
├─────────────────────────────┤
│   Tensor / Backend Runtime   │
└─────────────────────────────┘

This separation allows the user-facing API to remain familiar while optimization decisions happen internally.

What Gets Cached?

Turbo-Torch does not simply cache every tensor produced by an operation.

Instead, the runtime can cache reusable execution metadata such as:

• operator signatures
• dispatch decisions
• compatible tensor layouts
• dtype/device resolution
• shape-dependent execution information
• Python-to-runtime conversion metadata

This makes the cache significantly lighter than storing complete tensor results.

Cache Invalidation

Cached execution paths are associated with the characteristics that produced them.

When those characteristics become incompatible, Turbo-Torch can invalidate or bypass the cached path.

For example:

Cached:
    matmul
    float32
    CUDA
    [1024, 1024]

New request:
    matmul
    float16
    CUDA
    [2048, 2048]

                ↓

       cache mismatch
                ↓
        resolve new path
                ↓
          cache update

This prevents stale execution metadata from being reused incorrectly.

Performance Philosophy

Turbo-Torch is designed around a simple principle:

Optimize the path to the operation, not just the operation itself.

For workloads dominated by large GPU kernels, the performance difference may be limited because the underlying kernel execution dominates total runtime.

Turbo-Torch is therefore particularly interested in workloads where Python-side dispatch and repeated runtime bookkeeping represent a meaningful portion of execution time.

API Compatibility

Turbo-Torch intentionally mirrors the PyTorch programming model wherever practical.

Typical imports can be adapted from:

import torch
import torch.nn as nn
import torch.optim as optim

to:

import turbo_torch as torch
import turbo_torch.nn as nn
import turbo_torch.optim as optim

The rest of the application can remain structurally similar.

Design Goals

• Familiar PyTorch-style API
• Low Python-level dispatch overhead
• Cython-assisted runtime paths
• Lightweight internal caching
• Shape/dtype/device-aware execution metadata
• Minimal changes to existing code
• Transparent cache invalidation
• Backend-agnostic optimization where possible

Project Status

Turbo-Torch is currently an experimental runtime layer / research project exploring whether transparent caching and Cython-assisted dispatch can reduce overhead in PyTorch-style workloads.

Performance characteristics depend heavily on workload, tensor sizes, backend, device, and execution pattern.

Benchmarks should therefore be performed against the specific workload rather than assuming a universal speedup.

License

This project is intended for experimentation and research.