DCT-Vision

Frequency-domain native image processing. Operates directly on JPEG DCT coefficients, skipping the pixel decode step entirely.

Why?

JPEG images are already stored as DCT coefficients. Decoding to pixels just to blur/sharpen/adjust is wasteful. DCT-Vision works directly on the coefficients -- many operations become simple multiplications instead of expensive convolutions.

What's new in 0.3.0

• Quality metrics (dct_vision.math.metrics): PSNR/SSIM/MSE, SSIM validated against scikit-image.
• Honest benchmarks: every operation now reports quality (PSNR/SSIM) and peak memory alongside speed, plus an end-to-end (load+op+save) comparison -- not just op-only timing.
• Lossless geometry: 90/180/270 rotation and transpose (dv rotate), bit-exact on all chroma subsampling modes.
• ML: DCT/pixel ResNet-18, in-memory DCT precompute cache, reproducible seeded augmentation, and a fair 3-way comparison (pixel-RGB vs DCT-Y-only vs DCT-YCbCr).
• Applications (dv apps): near-duplicate detection, JPEG double-compression forensics, instant DC-only thumbnails.
• Fixes: 8x brightness DC-offset calibration, grayscale writer level-shift, downscale ~5x faster (BLAS matmul).

Full history: CHANGELOG.md. Detailed results and methodology: benchmarks/results/RESULTS.md.

Findings: where DCT wins and where it loses

The honest bottom line from the measurements (full results):

DCT wins

• Pointwise ops vs Pillow (brightness, contrast, sharpen, blur, edges): 12-19x faster.
• Brightness / contrast / noise vs OpenCV: several-fold faster (these dodge OpenCV's SIMD strengths).
• Exact, lossless transforms: brightness, flips, crop, 90/180/270 rotation, transpose -- zero quality loss, no IDCT.
• End-to-end on expensive ops and large images, and in batch pipelines that would otherwise decode every image.
• Frequency-native tasks where the signal is in the coefficients: forensics, thumbnails, perceptual-hash dedup are fast and natural.

DCT loses (or ties)

• vs OpenCV's SIMD on cheap geometric ops (flips, rotate, crop) and resampling (downscale) and edges -- OpenCV is highly optimized here.
• End-to-end on cheap ops at small sizes: libjpeg-turbo decode is already fast, so skipping it saves little.
• Approximate ops (blur, downscale, edges) trade some quality vs their spatial equivalents (see PSNR/SSIM below).
• Not translation-invariant: a sub-block pixel shift changes all coefficients -- a real limitation for some tasks.

ML finding (ML results): DCT input trains ~7-8x faster (the network sees a 4x4 block grid, not a 32x32 image) but does not match pixel accuracy on CIFAR-10. Color (YCbCr) recovers 4-7 points over Y-only; the residual gap is CIFAR's tiny size after 8x block downsampling, not a flaw in the representation. Speed and the accuracy cost are two ends of the same dial.

Performance

Full methodology and raw numbers: benchmarks/results/RESULTS.md (100 reps, resolutions 256-2048, JPEG quality 50-95). Reproduce with make bench.

Operation-only speed and quality (averaged over all resolutions/qualities). PSNR/SSIM compare the DCT op against the same operation applied in the spatial domain, so they measure the operation's own error. inf/1.000 means the transform is exact (lossless).

End-to-end (load + operation + save vs decode + operation + encode) is the fairer headline, since the whole premise is skipping the decode. DCT writes modified coefficients back losslessly:

Honest summary

• Big, consistent wins vs Pillow on pointwise ops (brightness, contrast, sharpen, blur, edges): 12-19x.
• vs OpenCV is mixed and we say so. Brightness/contrast/noise beat OpenCV several-fold; blur is about par; OpenCV's SIMD wins on flips, rotate, crop, downscale and edges. We do not claim to beat OpenCV at everything.
• Exact operations (brightness, flips, crop, and 90/180/270 rotation / transpose, all subsampling modes) are provably lossless. Blur/downscale/edges are approximations with the quality shown above.
• End-to-end favors DCT on expensive ops and large images; on cheap ops it is roughly par, because libjpeg-turbo decode is already fast, so there is less to save. The advantage compounds in batch pipelines that avoid repeated decode.

Install

pip install dct-vision

Quick start

Python API

from dct_vision.core.dct_image import DCTImage
from dct_vision.ops.blur import blur
from dct_vision.ops.color import adjust_brightness

# Load JPEG (extracts DCT coefficients directly, no pixel decode)
img = DCTImage.from_file("photo.jpg")

# Process in frequency domain
img = blur(img, sigma=2.0)
img = adjust_brightness(img, offset=20)

# Save (writes coefficients directly, no pixel encode)
img.save("output.jpg")

CLI

dv blur photo.jpg -o blurred.jpg --sigma 2.0
dv sharpen photo.jpg -o sharp.jpg --amount 1.5
dv brightness photo.jpg -o bright.jpg --offset 30
dv contrast photo.jpg -o contrast.jpg --factor 1.5
dv downscale photo.jpg -o small.jpg --factor 2
dv edges photo.jpg -o edges.jpg --method laplacian
dv rotate photo.jpg -o rotated.jpg --degrees 90     # lossless
dv info photo.jpg --json
dv quality photo.jpg
dv convert input.png -o output.jpg --quality 85
dv augment photo.jpg -o aug.jpg --flip horizontal --noise 3.0 --seed 42

# Applications
dv apps thumbnail photo.jpg -o thumb.jpg --size 64  # from DC coeffs, no IDCT
dv apps dedup ./photos/ --max-distance 5            # find near-duplicates
dv apps forensics photo.jpg                         # double-compression check

ML Augmentation Pipeline

from dct_vision.core.dct_image import DCTImage
from dct_vision.augment.flip import horizontal_flip
from dct_vision.augment.jitter import brightness_jitter
from dct_vision.augment.noise import gaussian_noise

img = DCTImage.from_file("train/img_001.jpg")
img = horizontal_flip(img)
img = brightness_jitter(img, max_offset=20, seed=42)
img = gaussian_noise(img, sigma=2.0, seed=42)
img.save("augmented/img_001.jpg")

Operations

PyTorch Integration

from dct_vision.ml.dataset import DCTDataset
from torch.utils.data import DataLoader

dataset = DCTDataset("train/", mode="y_only", resize_blocks=(4, 4),
                     augmentations=["hflip:p=0.5", "noise:sigma=2.0"])
loader = DataLoader(dataset, batch_size=32, num_workers=4)

Pre-cache for instant loading:

dv dataset prepare ./train/ -o ./train_dct/
dv dataset bench ./train/ --mode y_only --batch-size 32

DCT-input classification (CIFAR-10)

DCT coefficients feed straight into a CNN/ResNet (Uber 2018 style). The network sees a 4x4 block grid instead of a 32x32 image, so it does far less spatial compute. Three inputs compared fairly (same architecture, same augmentation): RGB pixels, DCT Y-only (grayscale), and DCT YCbCr (full color).

Reading it honestly: DCT input gives a large, consistent training speedup (~7-8x) but does not match pixel accuracy on CIFAR-10. Adding color (YCbCr) recovers 4-7 points over Y-only, confirming that part of the earlier Y-only gap was a color handicap, not the representation. The remaining gap is expected: CIFAR images are only 32x32, so DCT's 8x block downsampling leaves a 4x4 grid and discards spatial detail. The speedup and the accuracy cost are two ends of the same dial (less spatial resolution = less compute). Uber's accuracy-parity result was on ImageNet, where 8x downsampling still leaves ample resolution; closing the gap here needs larger inputs (STL-10/ImageNet), which the harness supports (--dataset stl10).

Applications

from dct_vision.apps import find_duplicates, detect_double_compression, dc_thumbnail
from dct_vision.core.dct_image import DCTImage

# Near-duplicate detection over a folder (pHash on coefficients, no decode)
groups = find_duplicates("./photos/", max_distance=5)

# Instant thumbnail from DC coefficients (one array op, no IDCT)
thumb = dc_thumbnail(DCTImage.from_file("photo.jpg"), size=64)

# JPEG double-compression forensics
report = detect_double_compression(DCTImage.from_file("suspect.jpg"))

Documentation

• Installation guide
• CLI reference
• Python API
• Architecture
• ML integration

Requirements

• Python 3.10+
• libjpeg-turbo (for native DCT extraction; falls back to Pillow if unavailable)

License

MIT