⚛️ Fuse augmentations

Write image augmentation as independent transforms — from Kornia, TorchVision, Albumentations, or plain numeric ranges. Execute compatible geometry with fewer resampling passes, and compatible color as one fused matrix or lookup table.

fuse-augmentations is a PyTorch tensor-first engine for reducing repeated interpolation in image augmentation pipelines. It recognizes a finite set of Kornia, TorchVision, and Albumentations transforms—or builds a pipeline directly from numeric ranges—then composes compatible transform matrices before pixels are sampled.

Animated WebP renders on GitHub and in any browser; macOS Preview and Finder Quick Look show only the first frame.

You keep the readable pipeline: rotate, scale, shear, translate, flip. The engine finds compatible runs and can replace several geometric warps with one.

[!IMPORTANT]

This package is Alpha and is not a general drop-in replacement for native Compose containers. It does not guarantee native pixels, input types, target processors, random streams, hooks, or universal speedups.

[!WARNING]

Use auxiliary targets only with explicitly supported spatial transforms. An unknown crop, resize, or other spatial passthrough can modify the image while leaving a mask, box, or keypoint tensor stale. Treat every Unknown ... SPATIAL_KERNEL barrier warning as unsafe when data_keys is present.

🔄 The problem: every warp resamples the image

A conventional chain may interpolate the same pixels after every geometric operation:

Native composite — each transform is its own warp (3 warps):

flowchart LR
  IN["Input<br/>image + annotations"]
  subgraph W1["Warp 1"]
    A1[Rotate]
  end
  subgraph W2["Warp 2"]
    A2[Translate]
  end
  subgraph W3["Warp 3"]
    A3[HFlip]
  end
  OUT["Output<br/>image + annotations"]
  IN --> W1 --> W2 --> W3 --> OUT
  classDef warp fill:#f7d0f7,stroke:#ff00ff,color:#000;
  classDef input fill:#ffff00,stroke:#cccc00,color:#000;
  class A1,A2,A3 warp;
  class IN input;
  style W1 fill:#d4edda,stroke:#00ff00;
  style W2 fill:#d4edda,stroke:#00ff00;
  style W3 fill:#d4edda,stroke:#00ff00;

Fused composite — the transforms collapse into one fused block (1 warp):

flowchart TB
  subgraph steps [" "]
    direction LR
    T1[Rotate] -.-> T2[Translate] -.-> T3[HFlip]
  end
  subgraph flow [" "]
    direction LR
    FIN["Input<br/>image + annotations"] --> FB["Fused block · warp 1<br/>M = M_hflip · M_trans · M_rot"] --> FOUT["Output<br/>image + annotations"]
  end
  steps --> FB
  classDef warp fill:#f7d0f7,stroke:#ff00ff,color:#000;
  classDef input fill:#ffff00,stroke:#cccc00,color:#000;
  class T1,T2,T3 warp;
  class FIN input;
  style steps fill:#ffffff,stroke:#ffffff;
  style flow fill:#ffffff,stroke:#ffffff;
  style FB fill:#d4edda,stroke:#00ff00,color:#000;

Repeated interpolation adds work, creates intermediate tensors, and can progressively discard high-frequency detail. Matrix composition is cheap by comparison: for a compatible run, the package samples the individual parameters, multiplies their homogeneous matrices in declared order, and evaluates the result through one sampling grid.

This does not mean the fused output is pixel-identical to the native chain. It is a different resampling strategy, and backend centers, fill rules, clipping, and interpolation conventions still matter.

Matched-parameter visual example

The same fixed Kornia rotation → scale → shear recipe is evaluated below. The native route resamples three times; Fuse Compose composes the geometry and samples once. The magenta/green/white overlay makes local disagreement visible without claiming native-pixel parity.

See three fixed recipes for each of Kornia, TorchVision, and Albumentations, including their exact limits.

✨ What the package can do

Built-in live-transform coverage

This is an allowlist, not a claim about every upstream transform. Rows are primitives named by the most common class name across backends; a check means the backend registers that class name, otherwise the cell shows the backend-specific class, entry point, or a note. Native/direct cells name the Compose.from_params arguments.

Geometry

Color

Per-channel non-linear scalar maps — gamma, solarize, posterize, and supported per-channel equalize — are registered per backend as listed in the Color table above. Static maps compose into a table; equalize builds a separate per-image, per-channel histogram table at its runtime position, so static neighbours remain fused on either side. The uint8 Albumentations native path and the TorchVision equalize table match their native sequential operations exactly; Kornia's installed equalize accepts floating tensors only. The float tensor path retains an interpolation tolerance: it detects large lookup jumps and switches to a sharp step rule there, rather than smearing a discontinuity across a grid cell. Residual non-linear barriers include cross-channel saturation/hue and Albumentations masked or luminance (by_channels=False) equalize, which are not per-channel scalar maps.

Other unknown and nonlinear operations generally become passthrough barriers. That preserves pipeline construction in many image-only cases, but passthrough is not automatically numerically transparent, device-efficient, or auxiliary-target safe.

Gaussian blur is a narrow exception: consecutive Gaussian blurs fold into one operation, and a Gaussian blur commutes to the end of the run when the affine that immediately follows it does not downscale, letting that affine run collapse to a single warp (the surrounding affines then fuse as any affine chain does). Axis-aligned affine scales use Kornia's native Gaussian primitive; rotated (or sheared-and-upscaled) affines use a normalized sampled full-covariance Gaussian with 3-sigma support capped at 63 pixels. A pure shear has a smallest singular value below one, so it is refused like any other downscale — only a shear combined with enough upscale to keep both singular values at or above one reaches the covariance path. The blur stays a barrier when the following affine downscales (its smallest singular value drops below one), and always for projective transforms and non-linear kernels. Kornia's installed sharpness remains a barrier because it clamps intermediate values and restores borders, so it is not a linear shift-invariant kernel chain. The native Albumentations Gaussian path folds consecutive blurs only; it does not commute through affines because that fused path cannot guarantee the backend's random-number stream.

📦 Install

pip install fuse-augmentations

The base package requires Python 3.10+ and PyTorch 2.2+. It includes the native/direct builder.

Install optional adapter ecosystems only when needed:

pip install "fuse-augmentations[kornia]"
pip install "fuse-augmentations[torchvision]"
pip install "fuse-augmentations[albumentations]"
pip install "fuse-augmentations[all]"

🚀 Quick start: no optional augmentation backend

import torch

from fuse_augmentations import Compose, ReorderPolicy

torch.manual_seed(7)

augment = Compose.from_params(
    rotation=(-15.0, 15.0),
    scale=(0.9, 1.1),
    shear_x=(-4.0, 4.0),
    translate_x=(-8.0, 8.0),
    hflip_p=0.5,
    reorder=ReorderPolicy.NONE,
)

images = torch.rand(4, 3, 128, 128, dtype=torch.float32)
augmented, matrix = augment(images, return_matrix=True)

assert augmented.shape == images.shape
assert augmented.dtype == images.dtype
assert matrix is not None
assert matrix.shape == (4, 3, 3)

print(augment.fusion_plan)
print(
    [
        (descriptor.kind, descriptor.n_warps_saved)
        for descriptor in augment.fusion_plan_descriptors
    ]
)

fused(_DirectParamTransform, _DirectFlipTransform)
[('fused', 1)]

The common fused input contract is a floating BCHW torch tensor. Both fuse_augmentations and the shorter fuse_aug import expose the same public objects.

🔌 Bring an existing backend pipeline

import torch
import torchvision.transforms.v2 as T

from fuse_augmentations import Compose, ReorderPolicy

augment = Compose(
    [
        T.RandomRotation(15.0),
        T.RandomAffine(degrees=0.0, scale=(0.9, 1.1)),
        T.RandomHorizontalFlip(p=0.5),
    ],
    reorder=ReorderPolicy.NONE,
)

output = augment(torch.rand(8, 3, 224, 224))

Kornia and Albumentations transform objects follow the same BCHW entry point when used in tensor pipelines. Albumentations also has an image-only HWC NumPy compatibility call, but it is not a replacement for native multi-target dictionaries and processors.

Mixed-backend pipelines are supported, with every backend change acting as a hard fusion boundary.

📊 What the measurements say

These numbers are from the 2026-07-12 local audit on macOS arm64, fuse-augmentations 0.9.0.dev0, Python 3.12, PyTorch 2.10, 256×256 inputs. They demonstrate the shape of the opportunity—not a release-wide promise.

Single operations can be slower because there is no resampling to eliminate. Color-heavy pipelines retain color cost. Small accelerator workloads can be dominated by launch, sampling, compilation, or conversion overhead. Always benchmark the exact production pipeline and keep a correctness/parity gate beside the timing.

🔬 Quality and semantics

Fusing geometry changes when interpolation happens. That can preserve more detail than repeatedly warping an already warped image, but it also means the result is not a byte-for-byte substitute for the native chain.

For research or parity-sensitive use:

• set ReorderPolicy.NONE explicitly;
• pair or record sampled parameters and matrices;
• compare output coordinates and task metrics, not only shapes;
• report backend, device, versions, dtype, image size, and batch size;
• separate compile warmup from steady-state timing;
• publish losses and skips alongside wins.

POINTWISE and AGGRESSIVE reordering are behavior-changing optimizations: moving color across geometry can alter border and clipped pixels. AGGRESSIVE currently behaves like POINTWISE.

🎯 Auxiliary targets: supported, but narrowly

Registered fused geometry can route:

• masks as BCHW tensors with nearest or bilinear sampling;
• boxes as dense (B, N, 4) xyxy or xywh tensors;
• keypoints as dense (B, N, 2) tensors.

Important boundaries:

• unknown spatial transforms may desynchronize image and targets;
• mask padding is always zero even when image padding is border/reflection;
• nearest masks are intentionally detached; bilinear requires floating soft masks;
• boxes are AABB-wrapped after rotation but are not clipped or filtered;
• labels, visibility, ragged instances, invalid-box removal, and keypoint validity are application responsibilities;
• the Albumentations HWC NumPy path is image-only.

For detection and segmentation, validate every transform class and warning before training.

🔍 Introspection without overreading it

• fusion_plan describes the current segment structure.
• fusion_plan_descriptors provides structured, serializable segment metadata.
• n_warps_saved is a plan estimate, not a literal native interpolation counter for exact operations.
• return_matrix=True and transform_matrix expose the last matrix-producing segment, not an automatic whole-pipeline matrix across backend, projective, crop, or passthrough boundaries.

Use per-call matrix return when output and transform provenance must stay paired.

Test-time de-augmentation

For one fused affine or projective geometric segment, pass the matrix returned by the same call to inverse to map a prediction back into the original frame. This pairing is safe for concurrent calls; inverse deliberately does not read the mutable transform_matrix property.

import torch

from fuse_augmentations import Compose

augment = Compose.from_params(translate_x=(2.0, 2.0))
images = torch.rand(1, 3, 16, 16)
prediction_augmented, matrix = augment(images, return_matrix=True)
prediction_original = augment.inverse(prediction_augmented, matrix=matrix)

assert prediction_original.shape == images.shape

With data_keys, pass the augmented auxiliary targets in the same positional order; masks use the matching sampling grid and boxes/keypoints use the inverse pixel matrix. Keypoints and masks recover to sampling precision, but bounding boxes are axis-aligned: a forward-then-inverse box is exact only for axis-aligned transforms (flip, scale, translation) and inflates under a rotation, shear, or projective warp. inverse raises instead of guessing for crop-resize (cropped pixels are lost), color/LUT/blur or passthrough segments, exact-only segments, multiple segments, or a missing paired matrix. It is geometric-only and cannot recover values discarded by interpolation or padding.

🎨 Synthetic datasets

Generate small labelled datasets of colored shapes — no training loop, no Lightning. Draw square/rectangle/triangle/circle in red/green/blue and export COCO or YOLO for detection, segmentation, or oriented bounding box (OBB).

import tempfile

from fuse_augmentations import generate_dataset

with tempfile.TemporaryDirectory() as out_dir:
    # COCO detection, 70/20/10 split, reproducible (pass a real path to keep it)
    counts = generate_dataset(
        out_dir, num_images=100, fmt="coco", task="detection", seed=0
    )

print(counts)

{'train': 70, 'val': 20, 'test': 10}

Swap fmt="yolo", task="obb", or class_mode="color" for other layouts, tasks, and class schemes.

rectangle plus random per-shape rotation give oriented boxes real orientation; all generation is seeded for byte-identical output. Generation streams — feed a training loop straight from SyntheticGenerator.generate(n) or a DataLoader via SyntheticIterableDataset, with no disk round-trip and bounded memory even for huge datasets. See the synthetic datasets guide and examples/generate_synthetic_dataset.py.

🧭 Where it fits

Use fuse-augmentations when:

• data is already a BCHW torch tensor;
• the pipeline contains several registered geometric transforms;
• repeated resampling is a fidelity, memory, or throughput concern;
• you can validate output semantics for the exact backend and task.

Prefer the native backend container when you require:

• PIL or unbatched CHW input;
• full Albumentations dictionary processors;
• exact native centers, fills, pixels, hooks, or RNG behavior;
• unsupported spatial transforms with masks, boxes, or keypoints;
• backend-specific per-transform interpolation semantics; per-transform border modes are available with padding_mode="per_transform" when they map exactly, while opaque modes stay native boundaries with a warning and the default override is unchanged.

📚 Documentation

The repository includes a complete MkDocs Material site:

• Overview
• Installation and backend-free quickstart
• How fusion works
• Exact capabilities
• Backend and configuration guides
• Auxiliary-target safety
• Reproducibility
• Quality and benchmark evidence
• Research methodology
• Known limitations
• FAQ
• Application walkthroughs
• generated references for the notable public API

The site configuration provides local search, per-page descriptions, canonical/Open Graph metadata, sitemap and crawler files, an llms.txt agent index, and GitHub Pages publication automation.

🧪 Reproduce the evidence

Benchmark and memory scripts live in experiments/. They expose cases where fusion loses as well as wins.

uv run --all-extras --group benchmark python experiments/optimize_score.py
uv run --all-extras --group benchmark python experiments/bench_gpu_batch.py --quick
uv run --all-extras --group benchmark python experiments/bench_memory.py --quick

These three headline commands are not the full set: experiments/ holds five benchmark scripts in total, including bench_augmentation_pipelines.py and bench_primitive_vs_affine.py, and the quality and benchmark evidence documentation page walks through all of them.

Treat quick runs as smoke evidence. Release-grade comparisons need independent processes, uncertainty intervals, paired RNG state, output-parity assertions, and full environment provenance.

🤝 Contributing

Bug reports and focused pull requests are welcome. Open an issue before a public API or architecture change.

Documentation example authoring and generated-test instructions are in CONTRIBUTING.md. Generated documentation tests are recreated in CI and should not be committed.

Build the docs locally with:

uv sync --group docs
uv run --group docs mkdocs build --strict

📄 License

Apache-2.0 © 2025–2026 Jiri Borovec.