linux-rt-upscaler 0.1.5

Real-time AI upscaler for any Linux window using CuNNy

Real‑Time Upscaler for Linux

A real‑time AI upscaler for any application window on GNU/Linux. It uses CuNNy neural networks to perform 2× upscaling, then scales the result to full screen while preserving aspect ratio. Mouse clicks and motion are automatically forwarded to the original window.

Motivation

While real-time upscaling tools like Magpie and Lossless Scaling remain Windows-exclusive, projects such as lsfg-vk are successfully bringing their frame generation capabilities to Linux. This project tackles the other half of the equation: AI-powered upscaling to deliver a native solution Linux has been missing, a fullscreen Gamescope-like experience that applies intelligent upscaling (similar to Anime4K) to any application.

Based on

Based on RealTimeSuperResolutionScreenUpscalerforLinux by L65536, with the following differences:

• Full‑screen scaling – The upscaled image now fills the monitor, applying a second scaling layer while preserving aspect ratio.
• Click/motion forwarding – Mouse clicks and motion are forwarded to the original window with proper coordinate transformation. This can be deactivated with the -d option.
• Support all CuNNy NVL models – All NVL models have been translated to pure HLSL to use.

Features

• AI‑Powered Upscaling – Uses the CuNNy (Convolutional upscaling Neural Network) models, trained specifically for high‑quality 2× upscaling of visual novels and illustrations .
• Complete Model Selection – Choose from all nine CuNNy NVL variants, offering a range of quality/performance trade‑offs:
  • 8x32 – Highest quality, slowest. Uses 8 internal convolutions with 32 feature layers.
  • 4x32 – Excellent quality, slightly faster than 8x32.
  • 4x24 – Balanced high quality with reduced layer size.
  • 4x16 – Good quality, moderate speed.
  • 4x12 – Lower quality, faster performance.
  • 3x12 – Reduced convolution count for better speed.
  • fast – Recommended for slower machines, good balance.
  • faster – Prioritizes speed over quality.
  • veryfast – Fastest option, lowest quality .
• Attach to Any Window – Either grab the currently active window, select from visible windows or launch a new program and capture its window automatically.
• Full‑Screen Output – The upscaled image is displayed in a transparent overlay that covers your entire monitor, scaled to fill the screen while preserving aspect ratio.
• Input Forwarding – Click, move, and drag on the upscaled image as if you were interacting directly with the original window.
• Hardware Accelerated – GPU compute via Compushady (Vulkan) works on NVIDIA, AMD, and Intel GPUs.
• Low Overhead – Minimal CPU/GPU usage; the final scaling pass uses hardware Lanczos2 filtering.

Requirements

• GNU/Linux with X11 (Wayland is not supported)
• Vulkan-capable GPU from any vendor (NVIDIA, AMD, Intel)
• Vulkan drivers (libvulkan-dev on Debian/Ubuntu)
• X11 development libraries (libx11-dev)
• Python 3.8 or newer

Installation

Option 1: Install from PyPI (Recommended)

# Install system dependencies (Debian/Ubuntu)
sudo apt install libvulkan-dev libx11-dev

# Install with pipx (recommended)
pipx install linux-rt-upscaler

# Or with regular pip
pip install linux-rt-upscaler

Option 2: Install from source

# Clone the repository
git clone https://github.com/baronsmv/linux-rt-upscaler.git
cd linux-rt-upscaler

# Install system dependencies (Debian/Ubuntu)
sudo apt install libvulkan-dev libx11-dev

# Install Python packages
pip install -r requirements.txt

# Install the package in development mode
pip install -e .

Usage

After installation, the upscale command will be available globally:

# Upscale the currently active window
upscale

# Interactively select from visible windows at startup
upscale -s

# Run a command and upscale its window
upscale <command>

# Choose a specific model (examples)
upscale -m 4x32      # High quality, balanced performance
upscale -m fast       # Recommended for slower machines
upscale -m veryfast   # Maximum performance

# Disable mouse‑click forwarding (also enables dimming/click‑through)
upscale -d

# Perform two 2× passes (total 4×) (Experimental)
upscale -2

# Show help and other options
upscale -h

Controls

• Exit: Press Ctrl+C in the terminal where the upscaler is running.
• Dimming/Click‑through (only when -d is used):
  • The overlay becomes semi‑transparent (20% opacity) when the mouse leaves the source window.
  • Clicks then pass through to whatever window is underneath (e.g., your desktop or other applications).

How It Works

1. Window Selection – Uses X11 to find the target window by PID or WM_CLASS.
2. Capture – Grabs the window's pixels using a fast custom C library.
3. AI Upscaling – CuNNy compute shaders (written in HLSL, compiled via Compushady) produce a 2× larger image .
4. Aspect‑Preserving Scaling – A lightweight Lanczos2 compute shader scales the upscaled image to fill the monitor, adding black bars to maintain the original aspect ratio.
5. Display – The result is rendered in a transparent overlay window that bypasses the window manager (so it always stays on top).
6. Input Forwarding – Mouse events are transformed using the scaling ratios and sent to the original window via XSendEvent.

Future Plans

• Addition of more models – Parse and include other models and shaders
• Standalone GUI application – Create a windowed app interface for easier management of the upscaler

Acknowledgments

• L65536 – For the original RealTimeSuperResolutionScreenUpscalerforLinux project, which provided the foundational scripts and CuNNy integration
• funnyplanter – For CuNNy, the neural network upscaling models, especially the NVL variants trained for visual novel content
• Compushady – Python library for GPU compute (Vulkan backend)
• PySide6 – Qt bindings used for the overlay window
• python‑xlib – X11 client library for window management and input forwarding
• pyewmh – Query and control of window manager
• psutil – Library for retrieving information on running processes