NeLux

NeLux is a high-performance Python library for video processing, leveraging the power of FFmpeg with hardware acceleration (NVDEC/NVENC). It delivers some of the fastest decode times globally, enabling efficient video decoding directly into ML-ready PyTorch tensors.

Originall created by Trentonom0r3

Installation

pip install nelux

Supported platforms:

PyTorch must be importable before nelux — the package uses torch's C++ runtime. For CUDA builds, install the matching CUDA torch wheel:

# Linux CUDA
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu132

# macOS / Linux CPU
pip install torch torchvision

Quick Start

Basic Usage

import torch  # must be imported before nelux
from nelux import VideoReader

# Open video with hardware acceleration (CPU path also supported)
reader = VideoReader("input.mp4", decode_accelerator="nvdec")

# Iterate frames — HWC uint8 by default (matches torchcodec convention)
for frame in reader:
    print(frame.shape)   # torch.Size([1080, 1920, 3]) — HWC
    print(frame.dtype)   # torch.uint8 for 8-bit sources; torch.int16 for >8-bit
                         # (override with force_8bit=True to always return uint8)

    # Permute to BCHW + cast to float when feeding to an ML model
    chw = frame.permute(2, 0, 1).unsqueeze(0).to(torch.float32) / 255.0
    output = model(chw)

Batch Frame Reading

import torch
from nelux import VideoReader

vr = VideoReader("video.mp4")

# Get specific frames — returned tensor is [B, H, W, 3] HWC uint8
batch = vr.get_batch([0, 10, 20])           # [3, H, W, 3]
batch = vr.get_batch_range(0, 100, 10)      # [10, H, W, 3]

# Pythonic slice / list notation (delegates to get_batch under the hood)
batch = vr[0:100:10]                        # [10, H, W, 3]
batch = vr[[-3, -2, -1]]                    # Last 3 frames (negative indexing OK)
single = vr[42]                             # Single frame [H, W, 3]

# Properties
print(len(vr))                              # Total frame count
print(vr.shape)                             # (frames, H, W, 3)

Motion Vectors

NeLux exposes the per-frame motion-vector side-data that FFmpeg's CPU decoders emit for inter-coded frames (P/B-frames). This is the raw macroblock / block-vector field the decoder used for prediction — useful for optical-flow pretraining, frame interpolation, video super-resolution, scene-cut detection, and codec-level diagnostics.

CPU decode only. decode_accelerator="nvdec" does not surface side-data, so both methods below require decode_accelerator="cpu". NVDEC/CUVID strips motion-vector export in exchange for the GPU throughput shown in the benchmarks above.

Preview generated by examples/motion_vector_overlay.py on a 640x360 H.264 clip — left is the raw frame, right overlays every 4th motion vector as a red arrow from the destination block toward its motion-compensated source.

import torch  # must be imported before nelux
from nelux import VideoReader

vr = VideoReader("video.mp4", decode_accelerator="cpu")

# Frame + vectors together — vectors is a list[dict] (one entry per block).
frame, vectors = vr.read_frame_with_motion_vectors()
for mv in vectors:
    print(mv["src_x"], mv["src_y"], "->", mv["dst_x"], mv["dst_y"])

# Vectors only — skips RGB conversion entirely; ~2-3x faster on inter frames.
# Returns (int32 [N, 10] array, frame_type) where frame_type is "I" | "P" | "B".
dense, frame_type = vr.read_motion_vectors()

Field schema. Each entry in vectors (and each row of the dense array) has these 10 columns, matching FFmpeg's AV_FRAME_DATA_MOTION_VECTORS:

To recover pixel-space displacement, divide motion_x/motion_y by motion_scale. I-frames (and codecs/decoder builds that don't export the side-data, e.g. some mpeg4 builds) return an empty list / zero-row array; frame_type lets you branch on that without inspecting the array.

Tip. read_motion_vectors() is the hot path when you only need the flow field — it skips libswscale RGB conversion and the host tensor copy, so on a typical 1080p P-frame it returns in a fraction of the read_frame() time.

Video Encoding

import torch
from nelux import VideoReader

reader = VideoReader("input.mp4")

# `create_encoder` pre-configures dimensions / fps / pixel format from the source.
with reader.create_encoder("output.mp4") as enc:
    for frame in reader:
        enc.encode_frame(frame)            # frame is [H, W, 3] uint8

print("Done!")

Carrying audio / subtitles across

add_passthrough copies (or transcodes) the source's audio and subtitle streams into the encoded output. Call it before the first encode_frame:

with reader.create_encoder("output.mp4") as enc:
    enc.add_passthrough("input.mp4")       # copy audio + subtitle streams
    for frame in reader:
        enc.encode_frame(frame)

# Trim a window + keep audio only (rebased to t=0):
reader.set_range(2.0, 6.0)                 # both float → seconds
with reader.create_encoder("clip.mp4") as enc:
    enc.add_passthrough("input.mp4", audio=True, subtitles=False, start=2.0, end=6.0)
    for frame in reader:
        enc.encode_frame(frame)

allow_transcode=True (default) re-encodes streams the output container can't stream-copy (e.g. AAC→WebM) instead of dropping them. One passthrough source per encoder; a second call raises.

Features

Core Features

• Hardware Acceleration: NVDEC (decode) and NVENC (encode) on NVIDIA GPUs
• Native HWC uint8 Output: frames decoded directly into a torch.Tensor of shape [H, W, 3] (or [H, W, 3] int16 for >8-bit sources; force_8bit=True clamps to uint8 always). No implicit float conversion — you cast/normalize on your side based on your model's expected input
• RGB or Grayscale Output: color_format="rgb" (default) or color_format="gray" for single-channel [H, W, 1] luma (libswscale BT.601/709-correct, not a channel average). CPU decode only. encode_frame also accepts single-channel input ([H, W, 1] or [H, W]). With a grayscale output pixel_format ("gray"/"gray16le") it's a verbatim, full-range data path — values stored exactly, up to true 16-bit, with a lossless (ffv1) round-trip — ideal for depth maps and masks; with a color pixel_format the gray input is replicated to RGB
• CPU Path Matches ffmpeg Byte-for-Byte: pure libswscale convert pipeline, default SWS_BILINEAR flags; output is bit-identical to ffmpeg -vf format=rgb24 on every common YUV/RGB format (see CHANGELOG v0.11.0)
• Batch Decoding: get_batch([...]) / vr[start:stop:step] returns [B, H, W, 3] with seek minimization, deduplication, and a dedicated random-access decoder
• Motion Vector Export: read_frame_with_motion_vectors() returns (frame, vectors) from FFmpeg decoder side-data; read_motion_vectors() skips RGB conversion and returns a dense [N, 10] array plus frame type. See preview + schema above and examples/motion_vector_overlay.py
• Audio / Subtitle Passthrough: encoder.add_passthrough(source, audio, subtitles, start, end) copies (or transcodes) audio + subtitle streams from a source into the output, with optional [start, end) trim + rebase to t=0

Performance Knobs

• prefetch=True: background producer thread (off by default — queue handoff costs ~2.5× more than the parallelism saves at typical decode speeds)
• convert_workers=N: explicit control over the CPU convert-pool size. None (default) uses min(hw_concurrency, 16) for throughput-max; 0 matches torchcodec's polite single-threaded convert footprint; positive N pins to that count. See CHANGELOG v0.11.0 for measured tradeoffs
• NVDEC fused convert: CUDA kernels for NV12 / P010 → RGB run in-line on the GPU; output stays on cuda:0 as a torch tensor — no CPU round-trip when decode_accelerator="nvdec"
• Decoder-side resize=(W, H): CPU path scales in libswscale; NVDEC uses cuvid's built-in resize=WxH — single pass, no post-decode F.interpolate/cv2.resize needed

Supported Codecs & Formats

CPU path supports anything libavcodec can decode (h264, hevc, vp8/9, av1, mpeg2/4, prores, …). NVDEC support depends on your GPU generation.

Benchmarks

H.264 decode → RGB tensor throughput, measured on Intel i9-13900K (24 logical cores) + RTX 3090, Windows 11, FFmpeg 8.x, PyTorch 2.11+cu130, nelux 0.11.0. Each row is the median of 5 fresh subprocess runs, 600 frames per run (300 at 4K). Output is HWC uint8 for every decoder (apples-to-apples).

Headline: nelux default vs torchcodec vs ffmpeg (CPU)

nelux fan-outs libswscale convert across cores → +14–67% fps over torchcodec at every res. The trade: ~2.5–3× CPU. RSS is essentially identical.

Polite mode (convert_workers=0) vs torchcodec

Disabling the convert worker pool matches torchcodec's single-threaded convert architecture exactly. fps + CPU + RSS land within ~2%:

So the "+14–67% fps" win above is entirely the convert worker pool — strip it and nelux ≈ torchcodec on every dimension. Pick the trade you want via convert_workers=N.

NVDEC (GPU decode) vs ffmpeg-nvdec

nelux NVDEC beats raw ffmpeg-nvdec by 8–32% on fps at lower CPU (NV12→RGB runs as a fused CUDA kernel; output stays on the GPU as a torch.Tensor, no host round-trip).

Quality (vs ffmpeg -vf format=rgb24 reference, 30-frame compare)

Across 14 (pix_fmt × colorspace) combos: 12 / 14 PSNR = ∞, SSIM = 1.000 — byte-identical to ffmpeg. The two exceptions are yuv420p10le (PSNR 47.9–48.3 dB / VMAF 99.85+) where 10→8-bit downconvert rounds differently from ffmpeg's direct 10-bit YUV→RGB path; perceptually identical. See tests/output/pixfmt_matrix/REPORT.md for the full table.

Caveats

• ffmpeg-rgb24 CPU% omitted — it runs as a subprocess; the psutil sampler ticks every 100 ms and ffmpeg startup is short, so the few samples it gets are not representative. fps is valid (time wall-clock).
• Single hardware data point — your numbers will differ. Reproduce with python tests/comprehensive_bench.py --tag mybox (full table) or python tests/bench_thread_modes.py (decoder-architecture comparison).
• Default prefetch=False matches typical use. With prefetch=True nelux can squeeze another ~3–5% fps on big clips but burns more RAM (background producer queue).

API Reference

VideoReader

VideoReader(
    input_path: str,
    num_threads: int = 0,                          # 0 = ffmpeg auto-detect
    force_8bit: bool = False,                      # cast >8-bit YUV down to uint8
    backend: Literal["pytorch", "numpy"] = "pytorch",
    decode_accelerator: Literal["cpu", "nvdec"] = "cpu",
    cuda_device_index: int = 0,                    # NVDEC GPU index
    resize: tuple[int, int] | None = None,         # decoder-side scale to (W, H)
    prefetch: bool = False,                        # background producer thread
    convert_workers: int | None = None,            # None = min(hw, 16); 0 = polite
    color_format: Literal["rgb", "gray"] = "rgb",  # "gray" = [H, W, 1] luma (CPU only)
)

Properties:

• width, height, fps, min_fps, max_fps, duration, total_frames
• pixel_format, bit_depth, aspect_ratio, codec, has_audio
• properties (full VideoProperties struct)
• shape → (frame_count, H, W, 3) (Python-side BatchMixin)
• frame_count → cached get_frame_count() (Python-side BatchMixin)

Methods:

• read_frame() / __next__() / iteration → next [H, W, 3] frame
• frame_at(timestamp: float | index: int) → random-access frame via secondary decoder (doesn't disturb iteration)
• __getitem__(int | float | slice | list | range) → single frame OR [B, H, W, 3] batch
• decode_batch(indices: list[int]) → C++ batch path; called by get_batch after validation
• get_batch(indices) / get_batch_range(start, end, step) → batch decode with seek minimization
• set_range(start, end) / reset() → bound iteration
• reconfigure(...) → reuse this VideoReader for a different file (10-50× faster than re-constructing)
• create_encoder(output_path) → VideoEncoder pre-configured to this source's dims/fps/format
• start_prefetch() / stop_prefetch() / prefetch_buffered / is_prefetching → runtime prefetch control
• supported_codecs() → list of codecs the linked libavcodec can decode

Documentation

• Full Usage Guide - Complete API reference
• Changelog - Version history
• Benchmarks - Performance comparisons

Requirements

• Python: 3.13+ (see pyproject.toml requires-python)
• PyTorch: 2.13+ (import torch must precede import nelux; the matching CUDA wheel provides the CUDA runtime nelux's NVDEC path needs)
• CUDA: 13.x (for NVDEC/NVENC builds). CPU-only builds drop this requirement.
• OS: Windows 10/11, Linux (manylinux_2_28+ / Ubuntu 22.04+), macOS 12+ (Apple Silicon, CPU only)

Building from Source

Build system is scikit-build-core + CMake + Ninja + vcpkg. There is no setup.py.

git clone https://github.com/NevermindNilas/NeLux.git
cd NeLux

# Editable install — invokes scikit-build-core, which configures CMake + Ninja
# and runs vcpkg under the hood. Set NELUX_ENABLE_CUDA=ON to build NVDEC/NVENC.
NELUX_ENABLE_CUDA=ON pip install -e .

# Or build a wheel
NELUX_ENABLE_CUDA=ON pip wheel . -w dist/

On Windows the build needs MSVC 18 (or compatible), and FFmpeg headers/libs under external/ffmpeg/ (see tools/download_ffmpeg.ps1).

See BUILD.md for detailed build instructions.

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). See the LICENSE file for details.

Acknowledgments

• FFmpeg: The backbone of video processing in NeLux
• PyTorch: For tensor operations and CUDA integration
• Contributors: Thanks to everyone who has contributed to NeLux!