This release is a pre-release and may not be stable for production use.
Hand the copies scattered through your code (copy_, gather / scatter, host ↔
GPU transfers, layout changes, casts) to copyengine. It merges them, decides
where to reorder and convert, and tunes itself to the hardware it runs on.
Existing code mostly stays as it is.
Before / after
# Before: layers × blocks small copies, each its own call (tens of thousands of calls)
for layer in range(L):
for blk in blocks:
scale_k[layer][blk].copy_(k_qs_buf[blk].expand(N, S))
# After: one line of wrapping. The loop is unchanged; its copies are merged
# into a few transfers and kernels.
with ce.scope():
for layer in range(L):
for blk in blocks:
scale_k[layer][blk].copy_(k_qs_buf[blk].expand(N, S))
Loops like this are usually rewritten by hand into a few vectorized copies or a custom kernel. copyengine aims to get that result without the rewrite, and without a new kernel for every new model or cache layout.
Where it helps
- Inference: loading and offloading KV cache blocks between host and GPU, paged caches (gather / scatter by block id), quantizing or dequantizing KV on the way (FP8 or int8 with a scale), swapping LoRA weights in and out.
- Training: offloading optimizer state and activations, dataloader H2D, checkpoint save / load and resharding (pure reordering).
- Model code:
x.permute(...).contiguous().cuda(), copying afterindex_select, permute-then-cast: write it as onece.copy_into the destination (withce.lazyfor gathers), and it happens in one pass, with the reordering done on the GPU.
copyengine moves and converts data. Computation such as reductions is out of scope: it runs as usual, ordered correctly around the copies.
What it guarantees
- Not slower than the original. Wherever torch has the same copy, torch's
own
copy_is always a candidate. Anything else is measured against it, including the time copyengine itself spends planning and submitting, and is kept only if it is clearly faster. Decisions keep being re-checked while your program runs. A scope as a whole is held to the same rule: each scope call site compares running intercepted with not intercepting at all (host and GPU time), and stops intercepting where that does not clearly win, for example around ops copyengine does not take (index_copy_) or copies torch already does best (stats()["scopes"]). The explicit API has the same protection where torch can express the copy:ce.lazyviews,ce.gather,ce.scatter_,ce.segmentsandce.indicesare measured against the same copy written with torch ops (indexing,index_copy_, slices) from the same host indices. Conversions withfn=and views torch has no form for (a reversed slice) have no torch equivalent: there copyengine picks the fastest of its own strategies. Either way it cannot beat torch code that does less work, for example indices already on the GPU, and each explicit call costs a few microseconds of host time: for a few small copies per step,freeze()or a CUDA graph is the fast path. - The same results. copyengine only takes a copy when it can produce
exactly torch's result; otherwise torch runs it. Host and GPU conversions
are bit-identical. (NaN stays NaN, but its sign and payload bits are not
kept: torch's own differ between its CPU and CUDA conversions.) This is a
guarantee about copies: moving and converting values. Quantization
(
ce.quantize) computes, and computation has no single "torch result": the order and form of the arithmetic decide the last bit, and torch itself gives different bits on the CPU and the GPU for the same expression. There copyengine follows its own written recipe (copyengine/_quant.py), with the same bits on the host and every GPU; torch code computing the same formula another way may differ in the last bit. - Easy to turn off.
CE_DISABLE=1turns copyengine's choices off: every copy runs through torch's owncopy_, or exactly as submitted where torch has no equivalent. - Bounded resources. With torch installed, copyengine's own device memory
(staging, copy tables) comes from torch's caching allocator, so it shows in
torch.cuda.memory_allocated()and obeys torch's limits; its pinned host buffers come from torch's pinned allocator (the onepin_memory()uses). Device staging has a budget (CE_STAGING_MB), andEngine(max_sms=N)(orCE_MAX_SMS) keeps copy kernels on at most N SMs per GPU, leaving the rest to your compute. Where the driver supports green contexts the limit is hard (checked on start); otherwise copyengine launches at most N thread blocks. DMA copies use the copy engines and no SMs; memory bandwidth is still shared, unless you limit it:ce.scope(bandwidth=8e9)(orEngine(bandwidth=...)) paces the copies on the GPU to about 8 GB/s per direction, leaving the rest of the link to other work.
The first runs of each new copy pattern are a warmup: copyengine measures its
options, then locks the fastest. Save the result with eng.save_profile(path)
to skip the warmup next time on the same machine.
Install
pip install copyengine
The wheels include the GPU kernels and the torch integration; they use the torch you already have. To build from source instead:
pip install . # needs CMake >= 3.26 and a C++20 compiler
# GPU support: CUDA toolkit headers + nvcc at build time
pip install --no-build-isolation . # same, with torch already installed: also builds the torch integration
dtypes: float64/32/16, bfloat16, float8 (e4m3fn, e5m2, e8m0fnu), int64/32/16/8,
uint8 and bool convert into each other; packed FP4 (float4_e2m1fn_x2) is
copied unconverted.
Requirements: Linux on x86_64 or aarch64 (server ARM such as Grace: GH200,
GB200, GB10; not Jetson), Python ≥ 3.10. GPU support runs on CUDA 12.x and
13.x drivers; the driver is loaded at runtime, nothing links against libcuda
or cudart.
The torch integration is built against the torch you have installed. If you
later switch torch versions, the part that intercepts copy_ is recompiled
once on import (needs a C++ compiler, takes seconds, cached). If that is not
possible, scopes fall back to a pure-Python interception with the same
results, only slower, and a warning says so.
Use torch 2.10 or newer for the best performance. Every torch from 2.9 on
gives the same results, but from 2.10 the explicit API (ce.copy_,
ce.lazy, ...) runs as a compiled torch op; on 2.9 it goes through Python and
costs a few more microseconds per call, which matters for many small copies.
scope() interception is compiled on every supported version.
Use
import copyengine as ce
import torch
# 1. Unchanged torch code: copy_ inside a scope is queued and merged.
with ce.scope():
for l in range(num_layers):
for i, b in enumerate(block_ids):
kv_cache[l, b].copy_(host_slot[l, i])
# 2. A copy torch cannot express as one view: gather + permute + cast in one pass.
src = ce.lazy(cache).index_select(0, ids).permute(0, 2, 1, 3)
ce.copy_(out_fp16, src)
# 3. Quantize / dequantize on the way.
ce.copy_(q_int8, x, fn=ce.Elementwise(scale=1 / qs, round=True, clamp=(-127, 127)))
# 4. Quantize while copying: FP8 / INT8 / INT4 / MXFP8 / NVFP4 per group of values.
ce.quantize(kv, q_cache, scales, fmt="fp8_e4m3", group=128, x_index=pages, index=slots)
# 5. Ragged or CSR layouts: many segments in one call.
ce.segments(src, dst, src_off, dst_off, length)
# 6. Overlap copies with compute: run them on a copy stream, the host never waits.
with ce.scope(stream="copy") as h: # or stream=<a torch.cuda.Stream>
for l in range(num_layers):
next_kv[l].copy_(host_kv[l, ids])
... # compute keeps running on the current stream
h.wait() # the current stream waits for the copies (GPU side)
# 7. A step that runs every iteration with new indices: plan it once.
def load_blocks(caches, pool, block_ids): # caches: one tensor per layer
for l, cache in enumerate(caches):
ce.gather(cache, block_ids, 0, out=pool[l])
step = ce.plan(load_blocks)
step(caches, pool, block_ids) # block_ids: a CUDA int64 tensor, new each step
ce.plan(step) (or eng.plan) traces step per argument layout until the
engine has settled, then replays its copies without running step or
planning. Index arguments are read anew on every call: CUDA int64 tensors
directly, and host indices (numpy arrays, lists of ints, CPU integer tensors)
when the step passes them straight to ce.gather, ce.scatter_ or
ce.lazy(...).index_select (indices the step computes from them are guarded
by value instead). For host indices two strategies are measured against each
other: DMA descriptors built at each call, and the statement kernel reading
the indices on the GPU. Every other tensor the copies touch must be an
argument and the same buffer (a new buffer or shape traces again). Only the copies replay, so a step should only
copy. A replay is kept only where it beats running step plainly.
eng.freeze() / plan.run() replays the last flush exactly (same buffers,
same indices).
More: ce.copy_many(dsts, srcs) for lists of copies (one per layer) in one
call, ce.gather / ce.scatter_ for index-based copies, ce.barrier() to
wait for queued work, ce.probe() to see which copy mechanisms this machine
supports (each verified by a real copy). C and C++ callers use
include/copyengine/copyengine.h (a stable C ABI) or copyengine::copyengine
from CMake.
To see what copyengine did: eng.stats()["last"]["decisions"] lists each
group of the last flush with its strategy and why (still exploring, the A/B
result it locked on, an online re-check, a CUDA graph capture), and
stats()["declined"] counts the intercepted copies it left to torch, by
reason. In Nsight Systems every group is an NVTX range named
copyengine:<strategy>.
To compare against torch on your GPU: python bench/bench_torch.py.
With an AI coding agent
Two Agent Skills teach coding agents (Claude Code, Codex, Cursor,
...) to adopt and debug copyengine: copyengine-adopt finds copy hotspots,
picks the integration, warms it up and proves same results and the measured
gain; copyengine-diagnose explains slow, declined or failing copies from
copyengine's own stats.
npx skills@1 add https://github.com/itdevwu/copyengine/tree/v0.3.0/skills/copyengine-adopt --global
npx skills@1 add https://github.com/itdevwu/copyengine/tree/v0.3.0/skills/copyengine-diagnose --global
How it works
Copies issued inside a scope are queued instead of run. When the scope ends,
copyengine looks at them together: which ones depend on each other (exact
address ranges), which can be merged across tensors, and what structure they
form (2-D blocks, strides, broadcasts). For each group it picks among merged
DMA calls, batched DMA (cuMemcpyBatchAsync), 2-D copies, copy kernels that
also cast and scale, moving data densely over the link and reordering it on the
GPU, or pinned bounce buffers for pageable memory, and measures which is
fastest.
Rules worth knowing
- Inside a scope, every other op is ordered after the queued copies whose
memory it reads or writes, so code in the scope sees the copied data.
Memory read without going through torch (
.numpy(), raw pointers) needsce.barrier()first. - GPU work runs on the current stream, like torch's own copies. GPU-to-GPU
copies are not waited for; intercepted
copy_keeps itsnon_blockingbehavior; the explicitce.copy_returns with any host-side copy complete (unless the engine was created withallow_async=True). - Threads:
ce.scope()andce.copy_use a per-thread default engine. AnEngineyou share between threads works while none of them has a scope open on it; a copy or scope from another thread during one raises an error. - Buffers of copies that are not waited for (intercepted
non_blockingcopies,allow_async) are kept alive until the GPU is done with them. - CUDA Graphs: copies inside
torch.cuda.graph(scopes,ce.copy_,FrozenPlan.run()) are recorded and replay correctly. Copies torch has no equivalent for must run once before capturing. scope(stream=...): the scope's work (its copies, and any other op written in it) runs on that stream, after what the current stream had queued. The host waits for none of it, even for copies that would otherwise block; the returned handle (wait(),synchronize(),done(),event) orders it against the rest. Its buffers stay alive until the copies are done. Inside a CUDA graph capture this is a fork, andh.wait()the join. Copies copyengine leaves to torch (for example a float-to-int cast) keep torch's own semantics.- Index tensors (
ce.lazy(...).index_select,ce.gather,ce.scatter_) may be host or CUDA tensors. CUDA indices stay on the GPU: the copy kernel reads them, so the host never waits and new indices reuse the same plan. An out-of-range CUDA index skips that element and raisesValueErrorat the nextsynchronize(); duplicate CUDA scatter indices are not detected (undefined, as in torch'sindex_copy_). - Each destination element must be written once per copy: duplicate scatter
indices and a source overlapping its own destination raise
UnsafeCopyin the explicit API (an interceptedcopy_leaves those to torch).
Environment
| Variable | Effect |
|---|---|
CE_DISABLE=1 |
turn copyengine's choices off: torch's own copy, or each copy as submitted |
CE_EXPLORE=0 |
never measure: use the loaded profile; copy patterns it does not cover run natively (or with copyengine's default where torch has no equivalent) and are listed in the profile for the next warmup |
CE_STAGING_MB |
device staging budget in MB per stream that stages copies (default 256) |
CE_LOG=plan |
print plans to stderr |
CE_SYNC=1 |
wait for every copy to finish (debugging) |
CE_MAX_SMS |
at most this many SMs per GPU for copy kernels (see guarantee 4) |
CE_SHIM=0 |
skip the compiled copy_ interception; scopes intercept in Python |
CE_TORCH_ALLOCATOR=0 |
take copyengine's device and pinned memory from cuMemAlloc / cuMemHostAlloc instead of torch's allocators |
CE_CACHE_DIR |
where interception builds compiled on import are cached (default ~/.cache/copyengine) |
Licensed under Apache-2.0.
Metadata
Release files for copyengine 0.4.0.dev2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| copyengine-0.4.0.dev2.tar.gz | 248.7 kB | With provenance | Details for copyengine-0.4.0.dev2.tar.gz |
Built distributions (wheels)
Total release size: 41.5 MB
Release files / copyengine-0.4.0.dev2.tar.gz
| Download URL | copyengine-0.4.0.dev2.tar.gz |
|---|---|
| Size | 248.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6fd1870e42ea0a80aed759d4648c15d715b3fede326822ea5fc87ef015d27ceb
|
|
BLAKE2b-256 checksum How to use checksums |
c735f4eede95eafe25ba10424085e9a2cbfea832ec60946fcaccf44684268014
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / copyengine-0.4.0.dev2-cp312-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
| Download URL | copyengine-0.4.0.dev2-cp312-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 6.9 MB |
| Tags | CPython 3.12 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
063856bbc84ef2ea00055790ec4feffafb0ea5331771bbbff8fad173f01349d5
|
|
BLAKE2b-256 checksum How to use checksums |
110a281c2ed33370ec4b622e413ee0e959e476e7ec3aad13c2806cb5f013a31f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / copyengine-0.4.0.dev2-cp312-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl
| Download URL | copyengine-0.4.0.dev2-cp312-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl |
|---|---|
| Size | 6.8 MB |
| Tags | CPython 3.12 Linux glibc 2.26+ ARM64 Linux glibc 2.28+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
097cc24785fa1a4a9ac3b27dc9c0e4198d482fd0cb677f265fce6a37264725aa
|
|
BLAKE2b-256 checksum How to use checksums |
aa3aa5894abae26354d1f1f66a05d15bd0f98c01bd7e1fad0ff77cdace8bd356
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / copyengine-0.4.0.dev2-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
| Download URL | copyengine-0.4.0.dev2-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 6.9 MB |
| Tags | CPython 3.11 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
cf47643e872a7ca5a2a1934b78e62a3191b629b768ca75b63741c7e44cdfc844
|
|
BLAKE2b-256 checksum How to use checksums |
29cb674fabc20cc2f39d4f6a7a3ee0fa7551f936eebb15045c08bc5e81496c0c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / copyengine-0.4.0.dev2-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl
| Download URL | copyengine-0.4.0.dev2-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl |
|---|---|
| Size | 6.8 MB |
| Tags | CPython 3.11 Linux glibc 2.26+ ARM64 Linux glibc 2.28+ ARM64 |
|
SHA-256 checksum How to use checksums |
9b3f7c7a9cff209cedb8f67258e45d9c215dc68e1b5d7f6c8441d706848984b8
|
|
BLAKE2b-256 checksum How to use checksums |
5c87bed23935341065ae70971465c1471b3b1b58298bcf30f9b2074eff42c338
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / copyengine-0.4.0.dev2-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
| Download URL | copyengine-0.4.0.dev2-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl |
|---|---|
| Size | 6.9 MB |
| Tags | CPython 3.10 Linux glibc 2.27+ x86-64 Linux glibc 2.28+ x86-64 |
|
SHA-256 checksum How to use checksums |
1aac3c36fd8812a589e3161e253720ed7d05d5de338c1d95d3339dc2900c9535
|
|
BLAKE2b-256 checksum How to use checksums |
f6cbc0a1254f251cac309c28d32b45920587de9983265d75f23900a38877b703
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logRelease files / copyengine-0.4.0.dev2-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl
| Download URL | copyengine-0.4.0.dev2-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl |
|---|---|
| Size | 6.8 MB |
| Tags | CPython 3.10 Linux glibc 2.26+ ARM64 Linux glibc 2.28+ ARM64 |
|
SHA-256 checksum How to use checksums |
5fb0380bbfd40aeccdc7dfc40ad1bc963836722b1e392eca79240b351c39d722
|
|
BLAKE2b-256 checksum How to use checksums |
1ede413a57e67500b84c0226c8ea9c4e7fcfee97edd418f1fdfacf9c0062fae2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logSecurity for copyengine 0.4.0.dev2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 7 files in this release have been uploaded with provenance metadata.
Attestation for copyengine-0.4.0.dev2.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 15, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.