Skip to main content

copyengine

Deliver optimal copy performance effortless.

Hand the copies scattered through your code (copy_, gather / scatter, host ↔ GPU transfers, layout changes, casts) to copyengine. It merges them, decides where to reorder and convert, and tunes itself to the hardware it runs on. Existing code mostly stays as it is.

Before / after

# Before: layers × blocks small copies, each its own call (tens of thousands of calls)
for layer in range(L):
    for blk in blocks:
        scale_k[layer][blk].copy_(k_qs_buf[blk].expand(N, S))

# After: one line of wrapping. The loop is unchanged; its copies are merged
# into a few transfers and kernels.
with ce.scope():
    for layer in range(L):
        for blk in blocks:
            scale_k[layer][blk].copy_(k_qs_buf[blk].expand(N, S))

Loops like this are usually rewritten by hand into a few vectorized copies or a custom kernel. copyengine aims to get that result without the rewrite, and without a new kernel for every new model or cache layout.

Where it helps

  • Inference: loading and offloading KV cache blocks between host and GPU, paged caches (gather / scatter by block id), quantizing or dequantizing KV on the way (int8 with a scale), swapping LoRA weights in and out.
  • Training: offloading optimizer state and activations, dataloader H2D, checkpoint save / load and resharding (pure reordering).
  • Model code: x.permute(...).contiguous().cuda(), copying after index_select, permute-then-cast: write it as one ce.copy_ into the destination (with ce.lazy for gathers), and it happens in one pass, with the reordering done on the GPU.

copyengine moves and converts data. Computation such as reductions is out of scope: it runs as usual, ordered correctly around the copies.

What it guarantees

  1. Not slower than the original. Wherever torch has the same copy, torch's own copy_ is always a candidate. Anything else is measured against it, including the time copyengine itself spends planning and submitting, and is kept only if it is clearly faster. Decisions keep being re-checked while your program runs. Copies only copyengine can express (ce.lazy gathers, ce.segments, conversions with fn=) have no original to compare with: there it picks the fastest of its own strategies, which can still lose to a torch rewrite of the same thing.
  2. The same results. copyengine only takes a copy when it can produce exactly torch's result; otherwise torch runs it. Host and GPU conversions are bit-identical.
  3. Easy to turn off. CE_DISABLE=1 turns copyengine's choices off: every copy runs through torch's own copy_, or exactly as submitted where torch has no equivalent.
  4. Bounded resources. Device staging memory has a budget (CE_STAGING_MB).

The first runs of each new copy pattern are a warmup: copyengine measures its options, then locks the fastest. Save the result with eng.save_profile(path) to skip the warmup next time on the same machine.

Install

pip install copyengine

The wheels include the GPU kernels and the torch integration; they use the torch you already have. To build from source instead:

pip install .                      # needs CMake >= 3.26 and a C++20 compiler
                                   # GPU support: CUDA toolkit headers + nvcc at build time
pip install --no-build-isolation . # same, with torch already installed: also builds the torch integration

Requirements: Linux x86_64, Python ≥ 3.10. GPU support runs on CUDA 12.x and 13.x drivers; the driver is loaded at runtime, nothing links against libcuda or cudart.

The torch integration is built against the torch you have installed. If you later switch torch versions, the part that intercepts copy_ is recompiled once on import (needs a C++ compiler, takes seconds, cached). If that is not possible, scopes fall back to a pure-Python interception with the same results, only slower, and a warning says so.

Use torch 2.10 or newer for the best performance. Every torch from 2.9 on gives the same results, but from 2.10 the explicit API (ce.copy_, ce.lazy, ...) runs as a compiled torch op; on 2.9 it goes through Python and costs a few more microseconds per call, which matters for many small copies. scope() interception is compiled on every supported version.

Use

import copyengine as ce
import torch

# 1. Unchanged torch code: copy_ inside a scope is queued and merged.
with ce.scope():
    for l in range(num_layers):
        for i, b in enumerate(block_ids):
            kv_cache[l, b].copy_(host_slot[l, i])

# 2. A copy torch cannot express as one view: gather + permute + cast in one pass.
src = ce.lazy(cache).index_select(0, ids).permute(0, 2, 1, 3)
ce.copy_(out_fp16, src)

# 3. Quantize / dequantize on the way.
ce.copy_(q_int8, x, fn=ce.Elementwise(scale=1 / qs, round=True, clamp=(-127, 127)))

# 4. Ragged or CSR layouts: many segments in one call.
ce.segments(src, dst, src_off, dst_off, length)

# 5. Replay a planned copy without planning again.
eng = ce.Engine()
with eng.scope():
    ...
plan = eng.freeze()
plan.run()

More: ce.gather / ce.scatter_ for index-based copies, ce.barrier() to wait for queued work, ce.probe() to see which copy mechanisms this machine supports (each verified by a real copy). C and C++ callers use include/copyengine/copyengine.h (a stable C ABI) or copyengine::copyengine from CMake.

To compare against torch on your GPU: python bench/bench_torch.py.

How it works

Copies issued inside a scope are queued instead of run. When the scope ends, copyengine looks at them together: which ones depend on each other (exact address ranges), which can be merged across tensors, and what structure they form (2-D blocks, strides, broadcasts). For each group it picks among merged DMA calls, batched DMA (cuMemcpyBatchAsync), 2-D copies, copy kernels that also cast and scale, moving data densely over the link and reordering it on the GPU, or pinned bounce buffers for pageable memory, and measures which is fastest.

Rules worth knowing

  • Inside a scope, every other op is ordered after the queued copies whose memory it reads or writes, so code in the scope sees the copied data. Memory read without going through torch (.numpy(), raw pointers) needs ce.barrier() first.
  • GPU work runs on the current stream, like torch's own copies. GPU-to-GPU copies are not waited for; intercepted copy_ keeps its non_blocking behavior; the explicit ce.copy_ returns with any host-side copy complete (unless the engine was created with allow_async=True).
  • Threads: ce.scope() and ce.copy_ use a per-thread default engine. An Engine you share between threads works while none of them has a scope open on it; a copy or scope from another thread during one raises an error.
  • Buffers of copies that are not waited for (intercepted non_blocking copies, allow_async) are kept alive until the GPU is done with them.
  • Each destination element must be written once per copy: duplicate scatter indices and a source overlapping its own destination raise UnsafeCopy in the explicit API (an intercepted copy_ leaves those to torch).

Environment

Variable Effect
CE_DISABLE=1 turn copyengine's choices off: torch's own copy, or each copy as submitted
CE_EXPLORE=0 never measure: use the loaded profile; copy patterns it does not cover run natively (or with copyengine's default where torch has no equivalent) and are listed in the profile for the next warmup
CE_STAGING_MB device staging budget in MB (default 256)
CE_LOG=plan print plans to stderr
CE_SYNC=1 wait for every copy to finish (debugging)
CE_SHIM=0 skip the compiled copy_ interception; scopes intercept in Python
CE_CACHE_DIR where interception builds compiled on import are cached (default ~/.cache/copyengine)

Licensed under Apache-2.0.

Metadata

Release files for copyengine 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for copyengine 0.1.0
File Size Uploaded Provenance Actions
copyengine-0.1.0.tar.gz 138.4 kB With provenance Details for copyengine-0.1.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for copyengine 0.1.0
File Interpreter ABI Platform Actions
copyengine-0.1.0-cp312-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl CPython 3.12 abi3 Linux glibc 2.27+ x86-64, Linux glibc 2.28+ x86-64 Details for copyengine-0.1.0-cp312-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
copyengine-0.1.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.27+ x86-64, Linux glibc 2.28+ x86-64 Details for copyengine-0.1.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
copyengine-0.1.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.28+ x86-64, Linux glibc 2.27+ x86-64 Details for copyengine-0.1.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl

Total release size: 13.9 MB

Release files / copyengine-0.1.0.tar.gz

Download URL copyengine-0.1.0.tar.gz
Size 138.4 kB
Tags Source
SHA-256 checksum
How to use checksums
8bbb7c1359aa5774a2bed9b58346425e1d374f1dd912b16e8b9ee18c7a3ccac7
BLAKE2b-256 checksum
How to use checksums
cff6e8bd0ce879c2d2d7bdcaf3eefa4a5c6a0481623d8a8befdc5463d669d65a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Release files / copyengine-0.1.0-cp312-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl

Download URL copyengine-0.1.0-cp312-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Size 4.6 MB
Tags CPython 3.12 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64 abi3
SHA-256 checksum
How to use checksums
a415c1bbe726bfafbc315c6ed32e12a258b3db5ab803725e7153aa5277ed68e7
BLAKE2b-256 checksum
How to use checksums
0fdf5a7dd296b8f5948b86ea5a6c7f0050ac66df9d1a9423dfcf1f4211474bac
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Release files / copyengine-0.1.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl

Download URL copyengine-0.1.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Size 4.6 MB
Tags CPython 3.11 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
8d138af441ebe5e529efbc95b3bffe368c3bf8ac55a5b93f30b436b14fe5e1b4
BLAKE2b-256 checksum
How to use checksums
e8d05f19d44a01ff76aac96fd3835ae0a255ead9016bd4ca826ad82fb05f9344
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Release files / copyengine-0.1.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl

Download URL copyengine-0.1.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
Size 4.6 MB
Tags CPython 3.10 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
ebab7446a033be9bb17b3e822063aeb4682d2817311a02c2866af2f3d6b2b3d1
BLAKE2b-256 checksum
How to use checksums
a507f4df84a3434d09feec31198e9f7c19c6c52a3d02742a30622e1685bdd3fc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Security for copyengine 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 4 files in this release have been uploaded with provenance metadata.

Select from 4 files

Attestation for copyengine-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page