object-remove 0.2.2

From-scratch reference implementation of a tiered on-device object-removal pipeline, CPU and GPU

object_remove

A from scratch implementation of an on device object removal pipeline (see docs/ARCHITECTURE.md for the full design): a polygon/scanline mask layer, four fill engines of increasing sophistication, two interchangeable blend backends, GPU acceleration (CUDA + Apple MPS) that engages automatically when available, and a locked/queued session persistence layer.

Everything here is implemented from primitives (FAST corners, BRIEF descriptors, Hamming matching, a quad-tree patch-grid solver, PatchMatch, Laplacian/Poisson blending) — no OpenCV, no learned weights.

Quick start

pip install object-remove

import numpy as np
from object_remove import RemoveObjectJob, EngineTier

result = RemoveObjectJob(image_rgb).run(mask, tier=EngineTier.AUTO,
                                        progress=lambda stage, frac: ...)
removed = result.image          # float32 RGB in [0, 1]
print(result.tier_used)         # which engine AUTO picked

image_rgb is any (H, W, 3) array (uint8 or float); mask is (H, W) with nonzero marking the object to remove. Every engine and blender defaults to device="auto": use a GPU (CUDA on NVIDIA, MPS on Apple Silicon) when one is available, otherwise the plain NumPy path — no separate install, no manual opt in. Pass "cuda"/"mps"/"cpu" to force a specific torch backend, or device=None to force plain NumPy regardless of hardware — see GPU acceleration below.

New to the package? Start with docs/USAGE.md — it walks through loading an image, building a mask three different ways (brush stroke, polygon selection, tap-to-auto-select), refining it, running the removal, and undo/session history, with runnable snippets for each.

To run from a source checkout instead of the installed package:

pip install -r requirements.txt          # numpy, scipy, Pillow, torch
PYTHONPATH=. python examples/demo.py     # writes before/after PNGs to examples/out/
PYTHONPATH=. python -m pytest -q         # tests

Module layout

Stage	Concern	This repo
A — mask	selection, brush, auto select	object_remove/mask/
B/D — scale/tiles	pyramid + tile scheduling	object_remove/pyramid/
C0 — legacy CPU fill	segment aware patch comparator	fillengines/patch_comparator.py
C1 — general solver	quad-tree patch-grid solve	fillengines/primary/
C2 — texture synthesis	self similar rigid shift	fillengines/self_similar_shift.py
C3 — patch match	multi scale PatchMatch	fillengines/patchmatch/
E — blend	multi band + Poisson	object_remove/render/
F — session	undo / persistence	object_remove/session/
— GPU runtime	device resolution	object_remove/runtime/device.py
— orchestration	tier selection, progress, cancellation	pipeline/remove_object_job.py

The four fill engines

• Tier 1 comparator — SegmentAwarePatchFillEngine — greedy onion peel exemplar fill scored by segment aware SSD with byte mask validity. Cheap, correct on small holes; the low end / tiny hole fallback.
• Tier 1 primary — PrimaryFillEngine — the general purpose solver: a quad-tree patch search (fillengines/primary/patch_search_tree.py) over a sparse grid of patch centres (not a dense per-pixel field), solved in onion peel order with adaptive, blur-tested scale selection instead of a fixed pyramid formula.
• Tier 2 — SelfSimilarShiftEngine — the fast default for repetitive texture. FAST corners + 256 bit BRIEF + Hamming matching find a rigid self similar offset; a 1D DP seam cuts the copied block in cleanly, run for x then y via a transpose trick.
• Tier 3 — PatchMatchEngine — multi scale, multi group PatchMatch inpainting (random init → propagation → random search, then patch voting reconstruction, coarse to fine). Available for explicit selection.

EngineTier.AUTO sends truly tiny holes to tier 1 comparator, otherwise tries the cheap tier 2 rigid shift first and falls back to tier 1 primary when the shift's boundary continuity score is poor.

GPU acceleration

torch ships as a regular dependency — one pip install object-remove gets both CPU and GPU support, nothing extra to install. Every fill engine and both blenders accept a device kwarg, defaulting to "auto": use a GPU (CUDA on NVIDIA, MPS on Apple Silicon) when one is available, otherwise the plain NumPy path. Explicit "cuda"/"mps"/"cpu" force that exact torch backend (and raise clearly if it isn't available); device=None forces plain NumPy regardless of hardware.

from object_remove.render.multiband_blender import MultiBandBlender
from object_remove.fillengines import PrimaryFillEngine

blender = MultiBandBlender(bands=3)          # device="auto" by default
engine = PrimaryFillEngine(device=None)      # force plain NumPy

Notes on the GPU path:

• The multi-band blender is a straight port (pure array math). The Poisson blender solves the identical linear system via matrix-free conjugate gradient instead of an exact sparse solve. The PatchMatch solver uses red-black checkerboard propagation instead of raster order, so GPU output legitimately differs pixel for pixel from the CPU path (same quality bar, different but equally valid traversal order).
• The torch path runs in float32 throughout (Apple MPS has weak/no float64 support) — a deliberate precision tradeoff vs. the plain NumPy path's float64.

The two blend backends

• MultiBandBlender — Laplacian pyramid blend with a fixed 4 tap kernel, [0.15439, 0.15133, 0.14252, 0.12896], and the mask feathered per pyramid level.
• PoissonBlend / PoissonBlend2 — gradient domain blend (sigma, multiplier, algo_n 1|2 variant), solved as a sparse Poisson system. Poisson is the safe default for single image inpainting because it reads only the (valid) hole boundary; the compositor repairs the background under the hole so no backend ever samples the object being removed.

Session persistence

SessionManager gives each edit a session id and an RLE mask history. Directory removal is dispatched onto a background queue and run under a lock — never an inline delete from the caller. A separate EmptySessionSweeper GCs orphaned empty session dirs left by aborted/crashed edits.

Beyond object removal

A handful of adjacent features share the same primitives as object removal rather than each reimplementing their own:

• object_remove/cv/ — reusable classic CV building blocks: ORB (FAST + BRIEF + Hamming, also used by tier 2), Harris/Shi-Tomasi corners, Canny edges, a distance transform, a separable max filter / non-max suppression, a generic sigma-configurable Gaussian/box blur, nearest-opaque fill, frequency separation.
• object_remove/effects/ — background_blur (selection-distance driven falloff, since no depth model ships) and FourPointPerspectiveCorrector (4-point homography + inverse warp).
• fillengines/clone_stamp.py — manual clone stamp: a user-specified source offset instead of a searched one, composing with PatchCompositor like any other engine.
• fillengines/wire_removal.py — wire/cable removal: a derived (Hessian ridge traced), not painted, mask acquisition model, feeding the same fill engines used for object removal.

Status & scope

Implemented and tested: the full pipeline across all stages, CPU (NumPy) and optional GPU (PyTorch: CUDA + Apple MPS) backends for every fill engine and blender, plus the adjacent features above. Auto select is explicitly an interface + CPU fallback: a real segmentation model can be dropped in behind AutoSelectModel, which ships a working color flood fallback so tap to select runs end to end without one.

object_remove/
  mask/        brush_rasterizer, polygon_selection, selection_enhancer,
               connected_components, auto_select
  pyramid/     image_pyramid, scale_scheduler, tile_scheduler, torch_pyramid
  fillengines/ patch_comparator (T1 comparator), primary/ (T1 general solver),
               self_similar_shift (T2), patchmatch/ (T3),
               clone_stamp, wire_removal
               each fill engine has a *_gpu.py / torch_*.py sibling module
  render/      multiband_blender(+_gpu), poisson_blender(+_gpu), compositor,
               fused_patch_blend
  cv/          orb, harris, canny, distance_transform, max_finder,
               symmetric_convolution, box_filter, nearest_opaque,
               frequency_separation
  effects/     background_blur, perspective_correction
  runtime/     device (GPU device resolution)
  session/     undo_session_manager, empty_session_sweeper
  pipeline/    remove_object_job