✨ Key Features

• 🚀 High Performance — C++ core releases the GIL during heavy computation for true multi-core scaling.
• 🔍 Ultra-Fast Probing — Extract JXL metadata (width, height, channels, bits) in < 0.05ms (~20μs) without decoding pixel buffers.
• 🖼️ Seamless Pillow Plugin — Transparent register_pillow() with true lazy loading (probe), full EXIF/ICC preservation, and fast saving.
• 🐍 Broad Python Compatibility — First-class support for Python 3.11, 3.12, 3.13, and Python 3.14.
• 📦 Metadata Excellence — Full support for EXIF, XMP, and JUMBF metadata, plus ICC color profile management.
• ⚡ Async-First & Backpressure — Native asyncio integration with double-layer semaphore backpressure for web services.
• 🎯 Elastic RunnerPool & Memory Pooling — Dynamic pool scaling, idle runner reaping, watermark buffer pooling, and timeout protection (CodecTimeoutError).
• 🖼️ NumPy Native & Zero-Copy — In-place decode (out=array) and Buffer Protocol support for zero memory allocation.
• 🔄 Lossless JPEG Transcoding — Bit-perfect JPEG ↔ JXL roundtrips without pixel decoding.

🛠️ Installation

Install from PyPI

# Recommended: Using uv
uv pip install pylibjxl

# Or via standard pip
pip install pylibjxl

Install from Source

uv pip install git+https://github.com/twn39/pylibjxl.git --recursive

Quick Start

🖼️ Basic In-Memory Operations

import numpy as np
import pylibjxl

# Create a test image (Height, Width, Channels)
image = np.random.randint(0, 256, (512, 512, 3), dtype=np.uint8)

# Encode to JXL bytes
data = pylibjxl.encode(image, effort=7, distance=1.0)

# Decode back to NumPy array
decoded = pylibjxl.decode(data)

💾 File I/O & Metadata

pylibjxl handles EXIF and XMP metadata seamlessly.

# Write an image with EXIF metadata
exif_data = b"Raw EXIF bytes..."
pylibjxl.write("output.jxl", image, effort=9, exif=exif_data)

# Read image and its metadata
img, meta = pylibjxl.read("output.jxl", metadata=True)
print(f"Loaded image shape: {img.shape}")
print(f"EXIF size: {len(meta.get('exif', b''))} bytes")

🔄 Lossless JPEG Transcoding

Reduce JPEG file size by ~20% without losing a single bit of information. The resulting .jxl can be restored to the exact original .jpg.

# Convert JPEG to JXL losslessly
pylibjxl.convert_jpeg_to_jxl("input.jpg", "input.jxl")

# Restore the bit-identical original JPEG
pylibjxl.convert_jxl_to_jpeg("input.jxl", "restored.jpg")

🔍 Ultra-Fast Header Probing (probe)

Extract JXL dimensions, channels, bit depth, and suggested threads in < 0.05ms (~20μs) without decoding pixel buffers or competing for worker threads in the runner pool:

# Probe raw bytes or memoryview
info = pylibjxl.probe(jxl_bytes)
print(info["width"], info["height"], info["channels"])

# Probe file directly (with optional zero-copy mmap)
info = pylibjxl.probe_file("image.jxl", use_mmap=True)

# Async variant for event loops
info = await pylibjxl.probe_async(jxl_bytes)

🖼️ Seamless Pillow Integration (register_pillow)

Seamlessly register pylibjxl as Pillow's official JXL codec engine. Enables standard Image.open() and im.save() with true lazy loading (via probe), full EXIF/ICC metadata preservation, and multi-core acceleration:

import pylibjxl
from PIL import Image

# Register transparently (idempotent, overrides legacy plugins by default)
pylibjxl.register_pillow()

# Open JXL image (dimensions read in ~20μs via probe, full decode deferred)
im = Image.open("photo.jxl")
print(im.size, im.mode, im.info.get("exif"), im.info.get("icc_profile"))

# Save as JXL with custom effort / quality / lossless parameters
im.save("output.jxl", effort=7, quality=90, lossless=False)

⚡ Async Support

High-performance non-blocking I/O for web servers and data pipelines.

import asyncio


async def main():
    # Async encoding
    data = await pylibjxl.encode_async(image, distance=0.0)

    # Async file reading
    img = await pylibjxl.read_async("input.jxl")


asyncio.run(main())

🏗️ Batch Processing (Context Manager)

Using the JXL context manager maintains a persistent thread pool, providing a significant speedup for batch operations.

# High-performance batch conversion
with pylibjxl.JXL(effort=7) as jxl:
    for i in range(100):
        img = jxl.read(f"input_{i}.jxl")
        # Process and save as high-quality JPEG
        jxl.write_jpeg(f"output_{i}.jpg", img, quality=95)

🚀 High-Concurrency Web Servers (FastAPI Example)

pylibjxl is engineered for high-concurrency production web servers and microservices. By combining double-layer backpressure with an elastic RunnerPool, it protects servers from memory exhaustion and CPU thrashing under traffic spikes.

from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException, UploadFile, status
import pylibjxl

# Allocate an isolated codec with a pool of up to 8 runners and a 3.0s timeout
codec = pylibjxl.AsyncJXL(pool_size=8, timeout=3.0)


@asynccontextmanager
async def lifespan(app: FastAPI):
    async with codec:
        yield


app = FastAPI(title="Image Service", lifespan=lifespan)


@app.post("/encode")
async def encode_image(file: UploadFile):
    content = await file.read()
    try:
        # Step 1: Decode JPEG (GIL released)
        image = await codec.decode_jpeg_async(content)

        # Step 2: Encode to JXL with per-operation timeout (e.g., 2.0s)
        # If all 8 runners are busy and cannot be acquired within 2.0s,
        # it raises CodecTimeoutError rather than queueing infinitely.
        jxl_bytes = await codec.encode_async(image, effort=5, timeout=2.0)
        return jxl_bytes

    except pylibjxl.CodecTimeoutError:
        # Gracefully handle server saturation
        raise HTTPException(
            status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
            detail="Image encoder is at maximum capacity; please retry shortly.",
        )

🧠 Concurrency & Resource Management (Deep Dive)

1. The Concurrency Challenge in libjxl

In the official Google libjxl architecture, JxlResizableParallelRunner is stateful and not thread-safe. A single runner instance cannot be shared across multiple threads or concurrent requests simultaneously.

2. The Throughput Formula: $M \times N \le \text{Cores}$

To maximize hardware utilization without causing CPU cache thrashing or context-switch storms, pylibjxl organizes concurrency according to the rule:

$$\text{Active Runners } (M) \times \text{Threads Per Runner } (N) \le \text{Available CPU Cores}$$

pylibjxl provides optimal presets for different workloads:

3. Elastic RunnerPool Lifecycle

• Zero Cold-Start Latency: Pre-allocates 1 warm runner upon creation so the first request incurs zero setup overhead.
• On-Demand Dynamic Expansion: Spawns additional runners only when concurrent load demands it, up to max_pool_size.
• Idle Reaping: Tracks timestamp activity and automatically deallocates idle runners beyond the baseline after idle_timeout (default 30s), reclaiming system memory during traffic lulls.

4. Double-Layer Backpressure

In Python web frameworks, asynchronous task queues (e.g. asyncio.to_thread) are unbounded by default. Under a traffic surge, thousands of coroutines can flood the thread pool queue, causing out-of-memory crashes.

AsyncJXL solves this with Double-Layer Backpressure:

1. Event Loop Layer (asyncio.Semaphore): Restricts the maximum number of concurrent tasks entering the thread pool. Excess tasks wait on the event loop with bounded asyncio.wait_for timeouts.
2. C++ Native Layer (std::condition_variable): Bounded acquire(timeout) ensures OS threads never block indefinitely.

# Check real-time pool metrics
print(f"Total runners: {codec.total_runners}")
print(f"Available runners: {codec.available_runners}")
print(f"In-use runners: {codec.in_use_runners}")

5. Timeout Protection & CodecTimeoutError

Both sync and async operations accept a timeout argument in seconds. When the runner pool cannot satisfy an acquisition request within the specified time, it raises pylibjxl.CodecTimeoutError.

CodecTimeoutError inherits from Python's standard TimeoutError, allowing catch-all handling:

try:
    data = await codec.encode_async(image, timeout=1.0)
except pylibjxl.CodecTimeoutError:
    # Specific codec timeout
    pass
except TimeoutError:
    # Standard Python timeout handler catches it too
    pass

📈 Performance & Stability

pylibjxl is engineered for high-throughput production environments.

🚀 pylibjxl vs. pillow-jxl Comparison

Tested on Apple Silicon (8-core), 1440x960 RGB image:

📊 Multi-Core Concurrency Throughput Scaling

Evaluated with pytest-benchmark on Apple Silicon (1440x960 RGB image):

🛠️ Key Architectural Advantages

• GIL-Free Execution: The C++ core releases Python's Global Interpreter Lock (GIL) during all heavy encoding and decoding tasks.
• Zero Memory Leaks: Verified over 500+ consecutive rounds with stable memory footprint.
• Zero-Copy Buffer Protocol: decode(data, out=ndarray) writes directly into pre-allocated NumPy memory with zero intermediate allocations.
• Event-Loop Purity: Array contiguity conversions (np.ascontiguousarray) execute inside worker threads to ensure 0ms blocking on the main asyncio event loop.

📂 API Reference

🖼️ JXL In-Memory Operations

encode(input, effort=7, distance=1.0, lossless=False, decoding_speed=0, *, exif=None, xmp=None, jumbf=None) -> bytes

async encode_async(...) -> bytes

Encodes a NumPy array into JPEG XL format.

# Synchronous encoding
data = pylibjxl.encode(image, effort=9, lossless=True)

# Asynchronous encoding
data = await pylibjxl.encode_async(image, distance=0.5)

decode(data, *, metadata=False) -> ndarray | tuple[ndarray, dict]

async decode_async(...) -> ndarray | tuple[ndarray, dict]

Decodes JPEG XL bytes back into a NumPy array.

# Basic decode
img = pylibjxl.decode(jxl_bytes)

# Decode with metadata
img, meta = await pylibjxl.decode_async(jxl_bytes, metadata=True)
print(f"EXIF size: {len(meta.get('exif', b''))} bytes")

💾 JXL File I/O

read(path, *, metadata=False) / async read_async(...)

Reads a .jxl file from disk and decodes it.

img = pylibjxl.read("input.jxl")
img, meta = await pylibjxl.read_async("input.jxl", metadata=True)

write(path, image, ...) / async write_async(...)

Encodes a NumPy array and writes it directly to a .jxl file.

pylibjxl.write("output.jxl", image, effort=7, distance=1.0)
await pylibjxl.write_async("output.jxl", image, lossless=True)

🔍 Ultra-Fast Image Probing

probe(data) -> dict / async probe_async(...) -> dict

Extracts image dimensions, channels, bit depth, and suggested threads in < 0.05ms (~20μs) without decoding pixel buffers or allocating runner threads.

Returns a dictionary:

{
    "width": 1920,
    "height": 1080,
    "channels": 3,
    "has_alpha": False,
    "bits_per_sample": 8,
    "suggested_threads": 2,
}

probe_file(path, *, use_mmap=True) -> dict

Probes image metadata directly from a file path.

info = pylibjxl.probe_file("photo.jxl", use_mmap=True)
print(f"{info['width']}x{info['height']}, {info['channels']} channels")

📷 JPEG Support (libjpeg-turbo)

encode_jpeg(input, quality=95) -> bytes / async encode_jpeg_async(...)

Encodes a NumPy array to JPEG bytes using high-speed libjpeg-turbo.

jpeg_bytes = pylibjxl.encode_jpeg(image, quality=90)

decode_jpeg(data) -> ndarray / async decode_jpeg_async(...)

Decodes JPEG bytes to a NumPy RGB array.

image = pylibjxl.decode_jpeg(jpeg_bytes)

read_jpeg(path) / write_jpeg(path, image, quality=95)

Stand-alone JPEG file I/O operations using libjpeg-turbo.

img = pylibjxl.read_jpeg("photo.jpg")
pylibjxl.write_jpeg("output.jpg", img, quality=85)

🔄 Lossless Transcoding (JPEG ↔ JXL)

jpeg_to_jxl(data, effort=7) -> bytes / async jpeg_to_jxl_async(...)

Transcodes raw JPEG bytes into a JPEG XL container losslessly.

jxl_data = pylibjxl.jpeg_to_jxl(jpeg_bytes)

jxl_to_jpeg(data) -> bytes / async jxl_to_jpeg_async(...)

Restores the original JPEG bytes from a transcoded JXL file.

original_jpeg = pylibjxl.jxl_to_jpeg(jxl_data)

convert_jpeg_to_jxl(in_path, out_path) / convert_jxl_to_jpeg(...)

File-to-file versions of the above transcoding operations.

pylibjxl.convert_jpeg_to_jxl("input.jpg", "output.jxl")
pylibjxl.convert_jxl_to_jpeg("output.jxl", "restored.jpg")

🏗️ Context Managers

JXL(effort=7, distance=1.0, lossless=False, decoding_speed=0, threads=0, pool_size=0, timeout=None, idle_timeout=30.0)

AsyncJXL(...)

Synchronous and Asynchronous context managers maintaining a private, elastic RunnerPool with double-layer backpressure and idle reaping.

Read-Only Telemetry Properties

Both JXL and AsyncJXL expose real-time metrics for health checks and observability:

• codec.pool_size (int): Maximum allowed runner instances.
• codec.total_runners (int): Total allocated runner instances currently in memory.
• codec.available_runners (int): Idle runners ready for immediate acquisition.
• codec.in_use_runners (int): Runners currently executing encoding/decoding tasks.
• codec.threads_per_runner (int): Number of internal worker threads per runner.

with pylibjxl.JXL(effort=7, pool_size=4, timeout=5.0) as jxl:
    img = jxl.read("input.jxl")
    jxl.write("output.jxl", img, distance=0.5)
    print(f"Active runners: {jxl.in_use_runners}/{jxl.total_runners}")

🖼️ Pillow Plugin Integration

register_pillow(*, override=True)

Registers pylibjxl as the JPEG XL (.jxl) image format plugin for Pillow.

• True Lazy Loading: Image.open() leverages probe to read dimensions instantly (< 0.05ms) without decoding pixel buffers until accessed.
• Comprehensive Modes: Supports RGB, RGBA, L (Grayscale), LA, and Palette (P) modes with automatic conversions.
• Metadata Roundtrip: Preserves and restores EXIF (im.info["exif"]) and ICC profiles (im.info["icc_profile"]).
• Idempotent: Safe to call repeatedly; override=True replaces any previously registered JXL handlers.

import pylibjxl
from PIL import Image

# Register plugin
pylibjxl.register_pillow()

# Standard Pillow operations
with Image.open("input.jxl") as im:
    print(im.size, im.mode)  # Lazy loaded, <0.05ms
    im_thumb = im.resize((128, 128))
    im_thumb.save("thumb.jxl", effort=7, quality=90)

ℹ️ System Information

print(f"pylibjxl version: {pylibjxl.version()}")

📜 License

BSD 3-Clause License