NeLux

NeLux is a high-performance Python library for video processing, leveraging the power of FFmpeg with hardware acceleration (NVDEC/NVENC). It delivers some of the fastest decode times globally, enabling efficient video decoding directly into ML-ready PyTorch tensors.

Originall created by Trentonom0r3

Installation

pip install nelux

Supported platforms:

FFmpeg ships inside the wheel — nothing needs to be installed or put on PATH. Every wheel carries the same build, TAS-FFMPEG 8.1.2, pinned by hash in tools/ffmpeg.lock and tagged so it is identifiable at runtime:

>>> nelux.__ffmpeg_version__
'8.1.2-tas'

If that reports anything else, a different FFmpeg of the same soname won the load — on Windows the first DLL of a given name into the process serves everyone, so another library shipping avcodec-62.dll can take over. ('unknown' is the exception, and it has two causes: the extension predates this attribute — rebuild it — or, on Windows, no FFmpeg could be loaded at all, which the extension reports instead of aborting the import.)

Those bundled binaries are GPL-2.0-or-later (libx264 and libx265 are linked in). The licence texts and a pointer to the complete corresponding source are installed at nelux/ffmpeg-licenses/ inside the package.

PyTorch must be importable before nelux — the package uses torch's C++ runtime. For CUDA builds, install the matching CUDA torch wheel:

# Linux CUDA
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu132

# macOS / Linux CPU
pip install torch torchvision

Quick Start

Basic Usage

import torch  # must be imported before nelux
from nelux import VideoReader

# Open video with hardware acceleration (CPU path also supported)
reader = VideoReader("input.mp4", decode_accelerator="nvdec")

# Iterate frames — HWC uint8 by default (matches torchcodec convention)
for frame in reader:
    print(frame.shape)   # torch.Size([1080, 1920, 3]) — HWC
    print(frame.dtype)   # torch.uint8 for 8-bit sources; torch.int16 for >8-bit
                         # (override with force_8bit=True to always return uint8)

    # Permute to BCHW + cast to float when feeding to an ML model
    chw = frame.permute(2, 0, 1).unsqueeze(0).to(torch.float32) / 255.0
    output = model(chw)

Batch Frame Reading

import torch
from nelux import VideoReader

vr = VideoReader("video.mp4")

# Get specific frames — returned tensor is [B, H, W, 3] HWC uint8
batch = vr.get_batch([0, 10, 20])           # [3, H, W, 3]
batch = vr.get_batch_range(0, 100, 10)      # [10, H, W, 3]

# Pythonic slice / list notation (delegates to get_batch under the hood)
batch = vr[0:100:10]                        # [10, H, W, 3]
batch = vr[[-3, -2, -1]]                    # Last 3 frames (negative indexing OK)
single = vr[42]                             # Single frame [H, W, 3]

# Properties
print(len(vr))                              # Total frame count
print(vr.shape)                             # (frames, H, W, channels)

In/Out Point Lists

set_ranges takes a list of (in, out) pairs, so one pass can cover several disjoint sections of a file — for applying different processing per section. iter_segments() tells you which section each frame came from.

from nelux import VideoReader

vr = VideoReader("video.mp4")

# Frame indices, seconds, or timecode strings — one unit throughout.
vr.set_ranges([(0, 1000), (5000, 6000)])
vr.set_ranges([("0:00:00", "2:00:00"), ("3:00:00", "4:00:00")])

for segment, frame in vr.iter_segments():
    out = grade_daylight(frame) if segment == 0 else grade_night(frame)

vr.ranges              # [(0.0, 7200.0), (10800.0, 14400.0)]
vr.clear_ranges()      # back to the whole file

Segments must be ascending and non-overlapping (touching is fine). Plain for frame in vr still yields bare frames, all segments back to back. See Multiple Segments for the seam and seek semantics.

Motion Vectors

NeLux exposes the per-frame motion-vector side-data that FFmpeg's CPU decoders emit for inter-coded frames (P/B-frames). This is the raw macroblock / block-vector field the decoder used for prediction — useful for optical-flow pretraining, frame interpolation, video super-resolution, scene-cut detection, and codec-level diagnostics.

CPU decode only. decode_accelerator="nvdec" does not surface side-data, so both methods below require decode_accelerator="cpu". NVDEC/CUVID strips motion-vector export in exchange for the GPU throughput shown in the benchmarks above.

Preview generated by examples/motion_vector_overlay.py on a 640x360 H.264 clip — left is the raw frame, right overlays every 4th motion vector as a red arrow from the destination block toward its motion-compensated source.

import torch  # must be imported before nelux
from nelux import VideoReader

# Motion-vector export costs decode time, so it is opt-in — enable it here.
vr = VideoReader("video.mp4", decode_accelerator="cpu", motion_vectors=True)

# The single motion-vector reader — vectors is a list[dict] (one per block).
frame, vectors = vr.read_frame_with_motion_vectors()
for mv in vectors:
    print(mv["src_x"], mv["src_y"], "->", mv["dst_x"], mv["dst_y"])

# The last frame's type ("I" | "P" | "B") is a separate property.
print(vr.frame_type)

Field schema. Each entry in vectors (and each row of the dense array) has these 10 columns, matching FFmpeg's AV_FRAME_DATA_MOTION_VECTORS:

To recover pixel-space displacement, divide motion_x/motion_y by motion_scale. I-frames (and codecs/decoder builds that don't export the side-data, e.g. some mpeg4 builds) return an empty list; the frame_type property lets you branch on that without inspecting the vectors.

Note. Motion-vector export is off by default because it makes the decoder compute per-block vectors on every frame (measurably slower, especially at 4K). Pass motion_vectors=True to VideoReader to enable it; the motion-vector reader raises a clear error otherwise.

Video Encoding

import torch
from nelux import VideoReader

reader = VideoReader("input.mp4")

# `create_encoder` pre-configures dimensions / fps / pixel format from the source.
with reader.create_encoder("output.mp4") as enc:
    for frame in reader:
        enc.encode_frame(frame)            # frame is [H, W, 3] uint8

print("Done!")

Carrying audio / subtitles across

add_passthrough copies (or transcodes) the source's audio and subtitle streams into the encoded output. Call it before the first encode_frame:

with reader.create_encoder("output.mp4") as enc:
    enc.add_passthrough("input.mp4")       # copy audio + subtitle streams
    for frame in reader:
        enc.encode_frame(frame)

# Trim a window + keep audio only (rebased to t=0):
reader.set_range(2.0, 6.0)                 # both float → seconds
with reader.create_encoder("clip.mp4") as enc:
    enc.add_passthrough("input.mp4", audio=True, subtitles=False, start=2.0, end=6.0)
    for frame in reader:
        enc.encode_frame(frame)

allow_transcode=True (default) re-encodes streams the output container can't stream-copy (e.g. AAC→WebM) instead of dropping them. One passthrough source per encoder; a second call raises.

Features

Core Features

• Hardware Acceleration: NVDEC (decode) and NVENC (encode) on NVIDIA GPUs
• Native HWC uint8 Output: frames decoded directly into a torch.Tensor of shape [H, W, C] (or uint16 for >8-bit sources; force_8bit=True clamps to uint8 always). C follows color_format and is readable as vr.channels: 3 for "rgb", 4 for "rgba", 1 for "gray". No implicit float conversion — you cast/normalize on your side based on your model's expected input
• RGB or Grayscale Output: color_format="rgb" (default) or color_format="gray" for single-channel [H, W, 1] luma (libswscale BT.601/709-correct, not a channel average). CPU decode only. encode_frame also accepts single-channel input ([H, W, 1] or [H, W]). With a grayscale output pixel_format ("gray"/"gray16le") it's a verbatim, full-range data path — values stored exactly, up to true 16-bit, with a lossless (ffv1) round-trip — ideal for depth maps and masks; with a color pixel_format the gray input is replicated to RGB
• CPU Path Matches ffmpeg Byte-for-Byte: pure libswscale convert pipeline, default SWS_BILINEAR flags; output is bit-identical to ffmpeg -vf format=rgb24 on every common YUV/RGB format (see CHANGELOG v0.11.0)
• Batch Decoding: get_batch([...]) / vr[start:stop:step] returns [B, H, W, C] with seek minimization, deduplication, and a dedicated random-access decoder
• Motion Vector Export (opt-in via motion_vectors=True): read_frame_with_motion_vectors() returns (frame, vectors) from FFmpeg decoder side-data; off by default so the common decode path stays fast. See preview + schema above and examples/motion_vector_overlay.py
• Audio / Subtitle Passthrough: encoder.add_passthrough(source, audio, subtitles, start, end) copies (or transcodes) audio + subtitle streams from a source into the output, with optional [start, end) trim + rebase to t=0
• In/Out Point Lists: set_ranges([(in, out), ...]) restricts iteration to several ascending, non-overlapping segments in one forward pass; iter_segments() yields (segment_index, frame) so each section can take its own processing path. Frames, seconds, or "H:MM:SS" timecodes

Performance Knobs

• prefetch=True: background producer thread (off by default — queue handoff costs ~2.5× more than the parallelism saves at typical decode speeds)
• convert_workers=N: explicit control over the CPU convert-pool size. None (default) uses min(hw_concurrency, 16) for throughput-max; 0 matches torchcodec's polite single-threaded convert footprint; positive N pins to that count. See CHANGELOG v0.11.0 for measured tradeoffs
• NVDEC fused convert: CUDA kernels for NV12 / P010 → RGB run in-line on the GPU; output stays on cuda:0 as a torch tensor — no CPU round-trip when decode_accelerator="nvdec"
• Decoder-side resize=(W, H): CPU path scales in libswscale; NVDEC uses cuvid's built-in resize=WxH — single pass, no post-decode F.interpolate/cv2.resize needed

Supported Codecs & Formats

CPU path supports anything libavcodec can decode (h264, hevc, vp8/9, av1, mpeg2/4, prores, …). NVDEC support depends on your GPU generation.

Benchmarks

H.264 decode → RGB tensor throughput, measured on Intel i9-13900K (24 logical cores) + RTX 3090, Windows 11, FFmpeg 8.x, PyTorch 2.11+cu130, nelux 0.11.0. Each row is the median of 5 fresh subprocess runs, 600 frames per run (300 at 4K). Output is HWC uint8 for every decoder (apples-to-apples).

Headline: nelux default vs torchcodec vs ffmpeg (CPU)

nelux fan-outs libswscale convert across cores → +14–67% fps over torchcodec at every res. The trade: ~2.5–3× CPU. RSS is essentially identical.

Polite mode (convert_workers=0) vs torchcodec

Disabling the convert worker pool matches torchcodec's single-threaded convert architecture exactly. fps + CPU + RSS land within ~2%:

So the "+14–67% fps" win above is entirely the convert worker pool — strip it and nelux ≈ torchcodec on every dimension. Pick the trade you want via convert_workers=N.

NVDEC (GPU decode) vs ffmpeg-nvdec

nelux NVDEC beats raw ffmpeg-nvdec by 8–32% on fps at lower CPU (NV12→RGB runs as a fused CUDA kernel; output stays on the GPU as a torch.Tensor, no host round-trip).

Quality (vs ffmpeg -vf format=rgb24 reference, 30-frame compare)

Across 14 (pix_fmt × colorspace) combos: 12 / 14 PSNR = ∞, SSIM = 1.000 — byte-identical to ffmpeg. The two exceptions are yuv420p10le (PSNR 47.9–48.3 dB / VMAF 99.85+) where 10→8-bit downconvert rounds differently from ffmpeg's direct 10-bit YUV→RGB path; perceptually identical. See tests/output/pixfmt_matrix/REPORT.md for the full table.

Caveats

• ffmpeg-rgb24 CPU% omitted — it runs as a subprocess; the psutil sampler ticks every 100 ms and ffmpeg startup is short, so the few samples it gets are not representative. fps is valid (time wall-clock).
• Single hardware data point — your numbers will differ. Reproduce with python tests/comprehensive_bench.py --tag mybox (full table) or python tests/bench_thread_modes.py (decoder-architecture comparison).
• Default prefetch=False matches typical use. With prefetch=True nelux can squeeze another ~3–5% fps on big clips but burns more RAM (background producer queue).

API Reference

VideoReader

VideoReader(
    input_path: str,
    num_threads: int = 0,                          # 0 = ffmpeg auto-detect
    force_8bit: bool = False,                      # cast >8-bit YUV down to uint8
    backend: Literal["pytorch", "numpy"] = "pytorch",
    decode_accelerator: Literal["cpu", "nvdec"] = "cpu",
    cuda_device_index: int = 0,                    # NVDEC GPU index
    resize: tuple[int, int] | None = None,         # decoder-side scale to (W, H)
    prefetch: bool = False,                        # background producer thread
    convert_workers: int | None = None,            # None = min(hw, 16); 0 = polite
    color_format: Literal["rgb", "gray"] = "rgb",  # "gray" = [H, W, 1] luma (CPU only)
)

Properties:

• width, height, fps, min_fps, max_fps, duration, total_frames
• pixel_format, bit_depth, channels, aspect_ratio, codec, has_audio
• properties (full VideoProperties struct)
• shape → (frame_count, H, W, channels) (Python-side BatchMixin)
• frame_count → cached get_frame_count() (Python-side BatchMixin)

Methods:

• read_frame() / __next__() / iteration → next [H, W, C] frame
• frame_at(timestamp: float | index: int) → random-access frame via secondary decoder (doesn't disturb iteration)
• __getitem__(int | float | slice | list | range) → single frame OR [B, H, W, C] batch
  • slices follow Python container rules: vr[:0] is empty, vr[:-1] drops the last frame, vr[-10:] is the last ten, vr[::-1] is reversed. A bound past the end raises IndexError rather than clamping
• decode_batch(indices: list[int]) → C++ batch path; called by get_batch after validation
• get_batch(indices) / get_batch_range(start, end, step) → batch decode with seek minimization
• set_range(start, end) / reset() → bound iteration (int frames, float seconds, or "H:MM:SS" timecode)
• set_ranges([(in, out), ...]) / clear_ranges() / ranges → several in/out segments in one pass
• iter_segments() → yields (segment_index, frame); current_segment for a plain loop
• reconfigure(...) → reuse this VideoReader for a different file (10-50× faster than re-constructing)
• create_encoder(output_path) → VideoEncoder pre-configured to this source's dims/fps/format
• start_prefetch() / stop_prefetch() / prefetch_buffered / is_prefetching → runtime prefetch control
• supported_codecs() → list of codecs the linked libavcodec can decode

Documentation

• Full Usage Guide - Complete API reference
• Changelog - Version history
• Benchmarks - Performance comparisons

Requirements

• Python: 3.13+ (see pyproject.toml requires-python)
• PyTorch: 2.13+ (import torch must precede import nelux; the matching CUDA wheel provides the CUDA runtime nelux's NVDEC path needs)
• CUDA: 13.x (for NVDEC/NVENC builds). CPU-only builds drop this requirement.
• GPU: compute capability 7.5+ (Turing, GTX 16xx / RTX 20xx and newer) for the published CUDA wheels. CUDA 13 dropped Pascal and Volta, so Maxwell/Pascal/Volta cards need a source build against CUDA 12.x (CUDAARCHS=61 pip install .) even though they have NVDEC silicon.
• OS: Windows 10/11, Linux (manylinux_2_28+ / Ubuntu 22.04+), macOS 12+ (Apple Silicon, CPU only)

Building from Source

Build system is scikit-build-core + CMake + Ninja + vcpkg. There is no setup.py.

git clone https://github.com/NevermindNilas/NeLux.git
cd NeLux

# Editable install — invokes scikit-build-core, which configures CMake + Ninja
# and runs vcpkg under the hood. Set NELUX_ENABLE_CUDA=ON to build NVDEC/NVENC.
NELUX_ENABLE_CUDA=ON pip install -e .

# Or build a wheel
NELUX_ENABLE_CUDA=ON pip wheel . -w dist/

FFmpeg comes from external/ffmpeg/, populated by tools/download_ffmpeg.ps1 (Windows) or tools/download_ffmpeg.sh (Linux/macOS). Both read tools/ffmpeg.lock, verify the archive's SHA256 and stamp the tree, so a build either has the exact pinned TAS-FFMPEG or fails loudly. On Windows the build also needs MSVC 18 (or compatible).

See BUILD.md for detailed build instructions.

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). See the LICENSE file for details.

Acknowledgments

• FFmpeg: The backbone of video processing in NeLux
• PyTorch: For tensor operations and CUDA integration
• Contributors: Thanks to everyone who has contributed to NeLux!