dngparser 0.1.0

Fast DNG parser for numpy datasets (DJI Pocket 4 + Apple ProRAW)

dngparser

Fast DNG (Digital Negative) parser for machine learning datasets. Written in C++ with pybind11 bindings, returns numpy arrays with zero torch dependency.

Supports multiple DNG vendors:

• DJI Pocket 4 — little-endian, uncompressed CFA Bayer (1ch uint16, strip-based)
• Apple ProRAW — big-endian, LJPEG-compressed LinearRaw (3ch uint16, tile-based)

Features

• C++ TIFF/DNG parser - reads TIFF container structure, IFD chains, SubIFDs, and all DNG-specific tags directly (no libtiff dependency)
• LJPEG decoder - built-in lossless JPEG (SOF3) decoder with 16-bit Huffman lookup tables for fast Apple ProRAW tile decompression
• Multi-format RAW read - uncompressed CFA Bayer (DJI) and LJPEG-compressed tiled LinearRaw (Apple), returned as numpy uint16 arrays
• Big-endian & little-endian - handles both TIFF byte orders with automatic conversion
• Full metadata extraction - ColorMatrix (SRATIONAL-aware), BlackLevel, WhiteLevel, AsShotNeutral, NoiseProfile, ActiveArea, LinearizationTable, exposure info, opcodes
• Numpy-first - load_dng() returns plain numpy arrays and Python primitives
• Type stubs - ships .pyi files generated by pybind11-stubgen for IDE autocomplete

Installation

pip install dngparser

Only depends on numpy. Install torch separately if you need it for dataset loading.

Quick Start

import dngparser

d = dngparser.load_dng("photo.dng")

# DJI Pocket 4 — CFA Bayer, 1 channel
raw = d["raw"]              # numpy.ndarray (H, W) uint16
print(d["cfa_pattern"])     # [0, 1, 1, 2] = RGGB
print(d["black_level"])     # [1024.0, 1024.0, 1024.0, 1024.0]
print(d["white_level"])     # 62400
print(d["compression"])     # 1 (uncompressed)

# Apple ProRAW — LinearRaw, 3 channels, LJPEG-compressed
d = dngparser.load_dng("IMG_5070.DNG")
raw = d["raw"]              # numpy.ndarray (H, W, 3) uint16
print(d["compression"])     # 7 (LJPEG)
print(d["bits_per_sample"]) # 12
print(d["is_tiled"])        # True
print(d["tile_width"])      # 504
print(d["linearization_table"])  # [0, 1, 2, ..., 4095]

# Shared metadata
print(d["color_matrix_1"])  # 3x3 float32, camera -> XYZ
print(d["as_shot_neutral"]) # [0.382, 1.0, 0.655]
print(d["noise_profile"])   # {"slope": 0.000258, "offset": 3.04e-08}

PyTorch Dataset Example

import random
import numpy as np
import torch
from torch.utils.data import Dataset
from dngparser import load_dng


class DngDataset(Dataset):
    """Paired RAW + JPG dataset for ISP training.

    Parameters
    ----------
    raw_dir : str or Path
        Directory containing .dng files.
    jpg_dir : str or Path, optional
        Directory containing corresponding .jpg files (same stem).
    crop_size : int, optional
        Random crop size for training. None = full resolution.
    normalize : bool
        If True, subtract black level and divide by white level.
    pack_bayer : bool
        If True, pack Bayer pattern into 4 channels (4, H/2, W/2).
    """

    def __init__(
        self,
        raw_dir,
        jpg_dir=None,
        crop_size=512,
        normalize=True,
        pack_bayer=True,
    ):
        from pathlib import Path

        self.raw_dir = Path(raw_dir)
        self.jpg_dir = Path(jpg_dir) if jpg_dir else None
        self.crop_size = crop_size
        self.normalize = normalize
        self.pack_bayer = pack_bayer
        self.raw_files = sorted(self.raw_dir.glob("*.dng"))

    def __len__(self):
        return len(self.raw_files)

    def __getitem__(self, idx):
        dng = load_dng(self.raw_files[idx])
        raw = torch.from_numpy(dng["raw"].copy())  # (H, W) uint16

        if self.normalize:
            raw = raw.to(torch.float32)
            black = torch.tensor(dng["black_level"], dtype=torch.float32)
            white = float(dng["white_level"])
            # Expand 2x2 black level to full image
            rows, cols = dng["cfa_repeat_rows"], dng["cfa_repeat_cols"]
            bl = black.view(rows, cols).repeat(
                raw.shape[0] // rows, raw.shape[1] // cols
            )[: raw.shape[0], : raw.shape[1]]
            raw = (raw - bl) / (white - bl.min().item())
            raw = raw.clamp(0.0, 1.0)
        else:
            raw = raw.to(torch.float32)

        if self.pack_bayer:
            # RGGB -> (4, H/2, W/2): [R, Gr, Gb, B]
            raw = torch.stack(
                [raw[0::2, 0::2], raw[0::2, 1::2],
                 raw[1::2, 0::2], raw[1::2, 1::2]],
                dim=0,
            )

        # Load JPG target if available
        if self.jpg_dir is not None:
            from PIL import Image

            jpg_path = self.jpg_dir / (self.raw_files[idx].stem + ".jpg")
            if jpg_path.exists():
                img = Image.open(jpg_path).convert("RGB")
                jpg = torch.from_numpy(np.array(img)).permute(2, 0, 1).float() / 255.0
            else:
                jpg = None
        else:
            jpg = None

        # Random crop
        if self.crop_size is not None and jpg is not None:
            h = raw.shape[-2]
            w = raw.shape[-1]
            y0 = random.randint(0, max(0, h - self.crop_size))
            x0 = random.randint(0, max(0, w - self.crop_size))
            raw = raw[..., y0:y0 + self.crop_size, x0:x0 + self.crop_size]
            jpg = jpg[..., y0:y0 + self.crop_size, x0:x0 + self.crop_size]

        result = {"raw": raw}
        if jpg is not None:
            result["jpg"] = jpg
        return result


# Usage
dataset = DngDataset(
    raw_dir="/path/to/dng_files",
    jpg_dir="/path/to/jpg_files",
    crop_size=512,
)
sample = dataset[0]
# sample["raw"]  -> (4, 512, 512) float32  (packed Bayer, normalized)
# sample["jpg"]  -> (3, 512, 512) float32  (sRGB target)

Vendor Differences

Feature	DJI Pocket 4	Apple ProRAW
Byte order	Little-endian (II)	Big-endian (MM)
Compression	Uncompressed (1)	LJPEG (7)
PhotometricInterpretation	CFA (32803)	LinearRaw (34892)
Channels	1 (Bayer)	3 (interleaved RGB)
Bit depth	16	12
Layout	Strips	Tiles (504x504)
LinearizationTable	absent	4096-entry LUT
NoiseProfile location	Main IFD	Raw SubIFD
WhiteLevel type	LONG	SHORT[3]
Raw output shape	(H, W) uint16	(H, W, 3) uint16

What It Does NOT Do

The parser only reads and structures data. It does not apply:

• Black level subtraction
• Linearization
• Demosaicing
• White balance / color correction
• Tone mapping

These transforms belong to the differentiable ISP pipeline (planned as a separate project).

Build from Source

Requires CMake 3.18+, a C++17 compiler, and Python 3.10+.

python -m venv .venv
.\.venv\Scripts\pip install scikit-build-core build pybind11 numpy

# Build wheel
.\.venv\Scripts\python.exe -m build --wheel

# Install
.\.venv\Scripts\pip install --force-reinstall --no-deps dist\dngparser-0.1.0-*.whl

# Test
.\.venv\Scripts\python.exe tests\test_dng_parse.py

Project Layout

cpp/
  include/dng/       # C++ headers: TIFF reader, LJPEG decoder, DNG parser, data structs
  src/               # C++ sources: tiff_reader.cpp, ljpeg_decoder.cpp, dng_parser.cpp
  binding/           # pybind11 module exposing load_dng()
python/dngparser/    # Python package: dng_loader.py, _dng.pyi
tests/               # End-to-end test with real DNG files (DJI + Apple)

License

MIT