NeLux

NeLux is a high-performance Python library for video processing, leveraging the power of FFmpeg with hardware acceleration (NVDEC/NVENC). It delivers some of the fastest decode times globally, enabling efficient video decoding directly into ML-ready PyTorch tensors.

Originall created by Trentonom0r3

Installation

pip install nelux

Supported platforms:

FFmpeg ships inside the wheel — nothing needs to be installed or put on PATH. Every wheel carries the same build, TAS-FFMPEG 8.1.2, pinned by hash in tools/ffmpeg.lock and tagged so it is identifiable at runtime:

>>> nelux.__ffmpeg_version__
'8.1.2-tas'

If that reports anything else, a different FFmpeg of the same soname won the load — on Windows the first DLL of a given name into the process serves everyone, so another library shipping avcodec-62.dll can take over. ('unknown' is the exception, and it has two causes: the extension predates this attribute — rebuild it — or, on Windows, no FFmpeg could be loaded at all, which the extension reports instead of aborting the import.)

Those bundled binaries are GPL-2.0-or-later (libx264 and libx265 are linked in). The licence texts and a pointer to the complete corresponding source are installed at nelux/ffmpeg-licenses/ inside the package.

PyTorch must be importable before nelux — the package uses torch's C++ runtime. For CUDA builds, install the matching CUDA torch wheel:

# Linux CUDA
pip install torch==2.14.0 torchvision==0.29.0 --index-url https://download.pytorch.org/whl/cu132

# macOS / Linux CPU
pip install torch==2.14.0 torchvision==0.29.0

Quick Start

Basic Usage

import torch  # must be imported before nelux
from nelux import VideoReader

# Open video with hardware acceleration (CPU path also supported)
reader = VideoReader("input.mp4", decode_accelerator="nvdec")

# Iterate frames — HWC uint8 by default (matches torchcodec convention)
for frame in reader:
    print(frame.shape)   # torch.Size([1080, 1920, 3]) — HWC
    print(frame.dtype)   # torch.uint8 for 8-bit sources; torch.uint16 for >8-bit
                         # (override with force_8bit=True to always return uint8)

    # Permute to BCHW + cast to float when feeding to an ML model
    chw = frame.permute(2, 0, 1).unsqueeze(0).to(torch.float32) / 255.0
    output = model(chw)

Batch Frame Reading

import torch
from nelux import VideoReader

vr = VideoReader("video.mp4")

# Get specific frames — returned tensor is [B, H, W, 3] HWC uint8
batch = vr.get_batch([0, 10, 20])           # [3, H, W, 3]
batch = vr.get_batch_range(0, 100, 10)      # [10, H, W, 3]

# Pythonic slice / list notation (delegates to get_batch under the hood)
batch = vr[0:100:10]                        # [10, H, W, 3]
batch = vr[[-3, -2, -1]]                    # Last 3 frames (negative indexing OK)
single = vr[42]                             # Single frame [H, W, 3]

# Properties
print(len(vr))                              # Total frame count
print(vr.shape)                             # (frames, H, W, channels)

In/Out Point Lists

set_ranges takes a list of (in, out) pairs, so one pass can cover several disjoint sections of a file — for applying different processing per section. iter_segments() tells you which section each frame came from.

from nelux import VideoReader

vr = VideoReader("video.mp4")

# Frame indices, seconds, or timecode strings — one unit throughout.
vr.set_ranges([(0, 1000), (5000, 6000)])
vr.set_ranges([("0:00:00", "2:00:00"), ("3:00:00", "4:00:00")])

for segment, frame in vr.iter_segments():
    out = grade_daylight(frame) if segment == 0 else grade_night(frame)

vr.ranges              # [(0.0, 7200.0), (10800.0, 14400.0)]
vr.clear_ranges()      # back to the whole file

Segments must be ascending and non-overlapping (touching is fine). Plain for frame in vr still yields bare frames, all segments back to back. Both units use [in, out): time outpoints are exclusive, with times relative to the first decoded frame. Frame ranges count decoded frames, including VFR; exact selection decodes through gaps, so large skipped portions take decode time. See Multiple Segments for the timing, EOF, and timestamp-availability rules.

Corpus validation covers reproducible reference checks, real-video holdout registration, and the installed-wheel CI gates.

Motion Vectors

NeLux exposes the per-frame motion-vector side-data that FFmpeg's CPU decoders emit for inter-coded frames (P/B-frames). This is the raw macroblock / block-vector field the decoder used for prediction — useful for optical-flow pretraining, frame interpolation, video super-resolution, scene-cut detection, and codec-level diagnostics.

CPU decode only. decode_accelerator="nvdec" does not surface side-data, so both methods below require decode_accelerator="cpu". NVDEC/CUVID strips motion-vector export in exchange for the GPU throughput shown in the benchmarks above.

Preview generated by examples/motion_vector_overlay.py on a 640x360 H.264 clip — left is the raw frame, right overlays every 4th motion vector as a red arrow from the destination block toward its motion-compensated source.

import torch  # must be imported before nelux
from nelux import VideoReader

# Motion-vector export costs decode time, so it is opt-in — enable it here.
vr = VideoReader("video.mp4", decode_accelerator="cpu", motion_vectors=True)

# The single motion-vector reader — vectors is a list[dict] (one per block).
frame, vectors = vr.read_frame_with_motion_vectors()
for mv in vectors:
    print(mv["src_x"], mv["src_y"], "->", mv["dst_x"], mv["dst_y"])

# The last frame's type ("I" | "P" | "B") is a separate property.
print(vr.frame_type)

Field schema. Each entry in vectors (and each row of the dense array) has these 10 columns, matching FFmpeg's AV_FRAME_DATA_MOTION_VECTORS:

To recover pixel-space displacement, divide motion_x/motion_y by motion_scale. I-frames (and codecs/decoder builds that don't export the side-data, e.g. some mpeg4 builds) return an empty list; the frame_type property lets you branch on that without inspecting the vectors.

Note. Motion-vector export is off by default because it makes the decoder compute per-block vectors on every frame (measurably slower, especially at 4K). Pass motion_vectors=True to VideoReader to enable it; the motion-vector reader raises a clear error otherwise.

Video Encoding

import torch
from nelux import VideoReader

reader = VideoReader("input.mp4")

# `create_encoder` pre-configures dimensions / fps / pixel format from the source.
with reader.create_encoder("output.mp4") as enc:
    for frame in reader:
        enc.encode_frame(frame)            # frame is [H, W, 3] uint8

print("Done!")

Carrying audio / subtitles across

add_passthrough copies (or transcodes) the source's audio and subtitle streams into the encoded output. Call it before the first encode_frame:

with reader.create_encoder("output.mp4") as enc:
    enc.add_passthrough("input.mp4")       # copy audio + subtitle streams
    for frame in reader:
        enc.encode_frame(frame)

# Trim a window + keep audio only (rebased to t=0):
reader.set_range(2.0, 6.0)                 # both float → seconds
with reader.create_encoder("clip.mp4") as enc:
    enc.add_passthrough("input.mp4", audio=True, subtitles=False, start=2.0, end=6.0)
    for frame in reader:
        enc.encode_frame(frame)

allow_transcode=True (default) re-encodes streams the output container can't stream-copy (e.g. AAC→WebM) instead of dropping them. One passthrough source per encoder; a second call raises.

Features

Core Features

• Hardware Acceleration: NVDEC (decode) and NVENC (encode) on NVIDIA GPUs
• Native HWC uint8 Output: frames decoded directly into a torch.Tensor of shape [H, W, C] (or uint16 for >8-bit sources; force_8bit=True clamps to uint8 always). C follows color_format and is readable as vr.channels: 3 for "rgb", 4 for "rgba", 1 for "gray". No implicit float conversion — you cast/normalize on your side based on your model's expected input
• RGB, RGBA or Grayscale Output: color_format="rgb" (default), "rgba" for [H, W, 4] carrying the source alpha plane (ProRes 4444 / 4444 XQ, VP9, PNG; straight alpha, opaque when the source has none — matching ffmpeg -pix_fmt rgba), or "gray" for single-channel [H, W, 1] luma (libswscale BT.601/709-correct, not a channel average). "rgba" and "gray" are CPU decode only. encode_frame also accepts single-channel input ([H, W, 1] or [H, W]). With a grayscale output pixel_format ("gray"/"gray16le") it's a verbatim, full-range data path — values stored exactly, up to true 16-bit, with a lossless (ffv1) round-trip — ideal for depth maps and masks; with a color pixel_format the gray input is replicated to RGB
• CPU Path Matches ffmpeg Byte-for-Byte: pure libswscale convert pipeline, default SWS_BILINEAR flags; output is bit-identical to ffmpeg -vf format=rgb24 on every common YUV/RGB format (see CHANGELOG v0.11.0)
• Batch Decoding: get_batch([...]) / vr[start:stop:step] returns [B, H, W, C] with seek minimization, deduplication, and a dedicated random-access decoder
• Motion Vector Export (opt-in via motion_vectors=True): read_frame_with_motion_vectors() returns (frame, vectors) from FFmpeg decoder side-data; off by default so the common decode path stays fast. See preview + schema above and examples/motion_vector_overlay.py
• Audio / Subtitle Passthrough: encoder.add_passthrough(source, audio, subtitles, start, end) copies (or transcodes) audio + subtitle streams from a source into the output, with optional [start, end) trim + rebase to t=0
• In/Out Point Lists: set_ranges([(in, out), ...]) restricts iteration to several ascending, non-overlapping segments in one forward pass; iter_segments() yields (segment_index, frame) so each section can take its own processing path. Frames, seconds, or "H:MM:SS" timecodes
• Encoder-Side Resize: VideoEncoder(..., resize=True, resize_filter=...) scales input frames to the output size inside the swscale pass the encoder already runs — one fused scale+convert, byte-identical to ffmpeg -vf scale=WxH:flags=bilinear, so no F.interpolate pass before encoding
• Decode-Free Metadata: nelux.probe(path) returns the full ffprobe-equivalent dict without opening a decoder, allocating a frame buffer or spawning threads — and without the subprocess spawn an ffprobe call pays

Performance Knobs

• prefetch=True: background producer thread (off by default — queue handoff costs ~2.5× more than the parallelism saves at typical decode speeds)
• convert_workers=N: explicit control over the CPU convert-pool size. None (default) uses min(hw_concurrency, 16) for throughput-max; 0 matches torchcodec's polite single-threaded convert footprint; positive N pins to that count. See CHANGELOG v0.11.0 for measured tradeoffs
• NVDEC fused convert: CUDA kernels for NV12 / P010 → RGB run in-line on the GPU; output stays on cuda:0 as a torch tensor — no CPU round-trip when decode_accelerator="nvdec"
• Decoder-side resize=(W, H): CPU path scales in libswscale; NVDEC uses cuvid's built-in resize=WxH — single pass, no post-decode F.interpolate/cv2.resize needed

Supported Codecs & Formats

CPU path supports anything libavcodec can decode (h264, hevc, vp8/9, av1, mpeg2/4, prores, …). NVDEC support depends on your GPU generation.

Benchmarks

H.264 decode → RGB tensor throughput, measured on Intel i9-13900K (24 logical cores) + RTX 3090, Windows 11, FFmpeg 8.x, PyTorch 2.11+cu130, nelux 0.11.0 — the last release these were re-measured on; the decode path has since gained work that shifts them (motion-vector gating in 0.16.0, the 0.17.0 batch-decode rewrite). Each row is the median of 5 fresh subprocess runs, 600 frames per run (300 at 4K). Output is HWC uint8 for every decoder (apples-to-apples).

Headline: nelux default vs torchcodec vs ffmpeg (CPU)

nelux fan-outs libswscale convert across cores → +14–67% fps over torchcodec at every res. The trade: ~2.5–3× CPU. RSS is essentially identical.

Polite mode (convert_workers=0) vs torchcodec

Disabling the convert worker pool matches torchcodec's single-threaded convert architecture exactly. fps + CPU + RSS land within ~2%:

So the "+14–67% fps" win above is entirely the convert worker pool — strip it and nelux ≈ torchcodec on every dimension. Pick the trade you want via convert_workers=N.

NVDEC (GPU decode) vs ffmpeg-nvdec

nelux NVDEC beats raw ffmpeg-nvdec by 8–32% on fps at lower CPU (NV12→RGB runs as a fused CUDA kernel; output stays on the GPU as a torch.Tensor, no host round-trip).

Quality (vs ffmpeg -vf format=rgb24 reference, 30-frame compare)

Across 14 (pix_fmt × colorspace) combos: 12 / 14 PSNR = ∞, SSIM = 1.000 — byte-identical to ffmpeg. The two exceptions are yuv420p10le (PSNR 47.9–48.3 dB / VMAF 99.85+) where 10→8-bit downconvert rounds differently from ffmpeg's direct 10-bit YUV→RGB path; perceptually identical. See tests/output/pixfmt_matrix/REPORT.md for the full table.

Caveats

• ffmpeg-rgb24 CPU% omitted — it runs as a subprocess; the psutil sampler ticks every 100 ms and ffmpeg startup is short, so the few samples it gets are not representative. fps is valid (time wall-clock).
• Single hardware data point — your numbers will differ. Reproduce with python tests/comprehensive_bench.py --tag mybox (full table) or python tests/bench_thread_modes.py (decoder-architecture comparison).
• Default prefetch=False matches typical use. With prefetch=True nelux can squeeze another ~3–5% fps on big clips but burns more RAM (background producer queue).

API Reference

VideoReader

VideoReader(
    input_path: str,
    num_threads: int = 0,                          # 0 = ffmpeg auto-detect
    force_8bit: bool = False,                      # cast >8-bit YUV down to uint8
    backend: Literal["pytorch", "numpy"] = "pytorch",
    decode_accelerator: Literal["cpu", "nvdec"] = "cpu",
    cuda_device_index: int = 0,                    # NVDEC GPU index
    resize: tuple[int, int] | None = None,         # decoder-side scale to (W, H)
    prefetch: bool = False,                        # background producer thread
    convert_workers: int | None = None,            # None = min(hw, 16); 0 = polite
    color_format: Literal["rgb", "gray", "rgba"] = "rgb",  # "gray" = [H, W, 1] luma,
                                                   #   "rgba" = [H, W, 4] with alpha
                                                   #   (both CPU decode only)
    resize_filter: str = "bilinear",               # sws kernel for resize, CPU only
    motion_vectors: bool = False,                  # opt-in MV export (costs decode time)
)

Properties:

• width, height, fps, min_fps, max_fps, duration, total_frames
• pixel_format, bit_depth, channels, aspect_ratio, codec, has_audio
• properties (full VideoProperties struct)
• shape → (frame_count, H, W, channels) (Python-side BatchMixin)
• frame_count → cached get_frame_count() (Python-side BatchMixin)

Methods:

• read_frame() / __next__() / iteration → next [H, W, C] frame
• frame_at(timestamp: float | index: int) → random-access frame via secondary decoder (doesn't disturb iteration)
• __getitem__(int | float | slice | list | range) → single frame OR [B, H, W, C] batch
  • slices follow Python container rules: vr[:0] is empty, vr[:-1] drops the last frame, vr[-10:] is the last ten, vr[::-1] is reversed. A bound past the end raises IndexError rather than clamping
• decode_batch(indices: list[int]) → C++ batch path; called by get_batch after validation
• get_batch(indices) / get_batch_range(start, end, step) → batch decode with seek minimization
• set_range(start, end) / reset() → bound iteration (int frames, float seconds, or "H:MM:SS" timecode)
• set_ranges([(in, out), ...]) / clear_ranges() / ranges → several in/out segments in one pass
• iter_segments() → yields (segment_index, frame); current_segment for a plain loop
• reconfigure(...) → reuse this VideoReader for a different file (10-50× faster than re-constructing)
• create_encoder(output_path) → VideoEncoder pre-configured to this source's dims/fps/format
• start_prefetch() / stop_prefetch() / prefetch_buffered / is_prefetching → runtime prefetch control
• supported_codecs() → list of codecs the linked libavcodec can decode

VideoEncoder

VideoEncoder(
    output_path: str,
    codec: str | None = None,          # container-aware default; see below
    width: int | None = None,          # 1920
    height: int | None = None,         # 1080
    bit_rate: int | None = None,       # 4_000_000
    fps: float | None = None,          # 30.0, tagged as an exact rational
    preset: int | str | None = None,   # 1..N per codec, or an ffmpeg preset name
    cq: int | None = None,             # constant quality, lower = better
    pixel_format: str | None = None,   # "yuv420p"
    options: dict[str, str] | None = None,  # extra AVOptions, applied last
    resize: bool = False,              # scale any input size to width x height
    resize_filter: str = "bilinear",   # sws kernel for that scale
)

The default codec is the first of libx264, libopenh264, then the platform encoder (h264_mf on Windows, h264_videotoolbox on macOS) that both exists in the build and fits the container inferred from output_path; failing that, the container's own default (.webm → VP9, .gif → gif, .ogv → VP8). Containers with no default this build can encode — .ogg, .wav — raise instead. fps is never rounded to an integer: 23.976 becomes 24000/1001, and the codec time base follows.

Methods:

• encode_frame(frame) → encode one [H, W, 3] RGB, [H, W, 4] RGBA, or [H, W, 1]/[H, W] grayscale tensor (CPU or CUDA). uint8 for 8-bit output; uint16/float [0,1] carry full precision into >8-bit formats
• add_passthrough(source, audio, subtitles, start, end, allow_transcode) → copy/transcode audio + subtitle streams, before the first frame
• close() / context manager → flush, write trailers, finalize
• is_hardware_encoder → True when NVENC is in use

Encoder-side resize

resize=True lets encode_frame take frames of any spatial size and scales them to width×height inside the libswscale pass the encoder already runs — one fused scale+convert, no separate resize stage, byte-identical to ffmpeg -vf scale=WxH:flags=bilinear:

with VideoEncoder("out.mp4", width=1920, height=1080, fps=30.0,
                  resize=True, resize_filter="lanczos") as enc:
    for frame in frames_2160p:        # (2160, 3840, 3) uint8
        enc.encode_frame(frame)

The input size locks to the first frame's shape (a later mismatch raises), and input must be an explicit HWC layout. A CUDA tensor at a different size takes the CPU staging path; same-size CUDA input keeps the zero-copy GPU path.

Module-level helpers

import nelux

nelux.probe("video.mp4")        # decode-free metadata dict — same keys as
                                #   VideoReader.properties, no decoder opened,
                                #   no frame buffer allocated, no threads spawned
nelux.get_available_encoders()  # [{"name", "long_name", "is_hardware"}, ...]
nelux.get_nvenc_encoders()      # NVENC-only subset
nelux.set_log_level(nelux.LogLevel.debug)
nelux.diagnose_runtime_dlls()   # Windows: report runtime DLLs that failed to load

nelux.__version__, nelux.__cuda_support__, nelux.__torch_abi__, nelux.__ffmpeg_version__

Documentation

• Full Usage Guide - Complete API reference
• Changelog - Version history
• Benchmarks - Performance comparisons

Requirements

• Python: 3.13+ (see pyproject.toml requires-python)
• PyTorch: 2.14.x for current 214torch wheels (import torch must precede import nelux; the matching CUDA wheel provides the CUDA runtime nelux's NVDEC path needs)
• CUDA: 13.x (for NVDEC/NVENC builds). CPU-only builds drop this requirement.
• GPU: compute capability 7.5+ (Turing, GTX 16xx / RTX 20xx and newer) for the published CUDA wheels. CUDA 13 dropped Pascal and Volta, so Maxwell/Pascal/Volta cards need a source build against CUDA 12.x (CUDAARCHS=61 pip install .) even though they have NVDEC silicon.
• OS: Windows 10/11, Linux (manylinux_2_28+ / Ubuntu 22.04+), macOS 14+ (Apple Silicon, CPU only)

Building from Source

Build system is scikit-build-core + CMake + Ninja + vcpkg. There is no setup.py.

git clone https://github.com/NevermindNilas/NeLux.git
cd NeLux

# Editable install — invokes scikit-build-core, which configures CMake + Ninja
# and runs vcpkg under the hood. Set NELUX_ENABLE_CUDA=ON to build NVDEC/NVENC.
NELUX_ENABLE_CUDA=ON pip install -e .

# Or build a wheel
NELUX_ENABLE_CUDA=ON pip wheel . -w dist/

FFmpeg comes from external/ffmpeg/, populated by tools/download_ffmpeg.ps1 (Windows) or tools/download_ffmpeg.sh (Linux/macOS). Both read tools/ffmpeg.lock, verify the archive's SHA256 and stamp the tree, so a build either has the exact pinned TAS-FFMPEG or fails loudly. On Windows the build also needs MSVC 18 (or compatible).

CMakePresets.json carries ready-made configure/build presets (x64-release-cuda, x64-release-cpu, and debug equivalents) if you would rather drive CMake directly. For Linux prerequisites — vcpkg, CMake and the system packages the build needs — see docs/linux.md; take its FFmpeg step from tools/download_ffmpeg.sh above rather than from vcpkg, which that page predates.

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0). See the LICENSE file for details.

Acknowledgments

• FFmpeg: The backbone of video processing in NeLux
• PyTorch: For tensor operations and CUDA integration
• Contributors: Thanks to everyone who has contributed to NeLux!