Add Temporal Coherence to AI Upscales in VapourSynth

When using SISR models (single image super resolution) on video, they tend to create slightly different results each frame. TemporalFix is a post filter that averages these results over multiple frames, removing temporal inconsistencies like fizzle or wiggly lines. Also known as stabilization, deshimmering, temporal denoising, or temporal fix.

Installation

Nvidia

pip install -U vs_temporalfix[tensorrt] --extra-index-url https://pypi.nvidia.com/

To enable the CPU/CUDA backends, install PyTorch with CUDA. (optional)

Others

pip install -U vs_temporalfix

To enable the CPU backend, install PyTorch. (optional)

TemporalFix AI Model

The newest and most capable version of temporalfix. It is easy to use and can run very fast on Nvidia GPUs.

import vs_temporalfix
clip = vs_temporalfix.model(clip, strength=2.0, exclude=None, backend="tensorrt", tiles=1, num_streams=1, gpu_id=0, engine_folder=None)

clip
Temporally unstable upscaled clip. Must be in RGBH format.

strength
Suppression strength of temporal inconsistencies in the 0.0-3.0 range. Higher means more aggressive.
Higher resolution tends to need higher strength. Too high may oversmooth small movements.

exclude (optional)
Optionally exclude scenes with intended temporal inconsistencies.
Brackets define excluded frame ranges. Example for two scenes: exclude="[10 20] [600 900]"

backend (optional)
The backend used to run the model:

• cpu CPU mode (very slow).
• cuda GPU mode using CUDA. Requires any Nvidia GPU (fast).
• directml GPU mode using DirectML. Works on most GPUs, but Windows only (faster, high vram).
• tensorrt GPU mode using TensorRT. Requires an Nvidia RTX GPU. On the first run, this mode will automatically build an engine, which may take a few minutes. Changing strength or input dimensions will trigger rebuilding, but previously build engines are stored (very fast, low vram).

tiles (optional)
A higher amount of tiles will reduce VRAM usage at the cost of speed.
This should only be needed on low end hardware. Default tiles=1 uses the full frame, which is fastest.

num_streams (optional)
Number of parallel TensorRT streams. For high end GPUs higher can be faster, but requires more VRAM. Only affects the TensorRT backend.

gpu_id (optional)
Which GPU to use starting from 0. Can be used to switch between iGPU/dGPU. Does not affect the CPU backend.

engine_folder (optional)
Optional path to the TensorRT engine storage location. By default engines are stored in vs_temporalfix/engines. Only affects the TensorRT backend.

TemporalFix Classic

The original CPU based version. It is harder to tune, may miss some areas, and only works well for 2D animation.

import vs_temporalfix
clip = vs_temporalfix.classic(clip, strength=500, tr=6, denoise=False, exclude=None, debug=False)

clip
Temporally unstable upscaled clip. Any format.

strength
Suppression strength of temporal inconsistencies. Higher means more aggressive. 400-700 works great in most cases.
The best way to finetune is to find a static scene and adjust till lines and details are stable.
Reduce if you get blending/ghosting on small movements, especially in dark or hazy scenes.

tr
Temporal radius sets the number of frames to average over.
Higher means more stable, especially on slow pans and zooms, but is slower. 6 works great in most cases.
The best way to finetune is to find a slow pan or zoom and adjust till lines and details are stable.

denoise (optional)
Removes grain and low frequency noise/flicker left over by the main processing step. Only enable if these issues actually exist! It risks to remove some details like every denoiser, but is useful if you're planning to denoise anyway and has the benefit of almost no performance impact compared to using an additional denoising filter.

exclude (optional)
Optionally exclude scenes with intended temporal inconsistencies.
Brackets define excluded frame ranges. Example for two scenes: exclude="[10 20] [600 900]"

debug (optional)
Shows areas that will not be fixed in pink. This includes areas with high motion, scene changes and excluded scenes. Can help while tuning parameters to see if the area is even affected.

Benchmarks

Model benchmarks were done on a RTX 4090 GPU and Classic benchmarks on a Ryzen 5900X CPU.

Third-Party Integrations

Several projects integrated TemporalFix to simplify usage without the need for VapourSynth knowledge. Feel free to contact me if you want to be part of this list.

• Vapourkit (Windows only)
  Video filter and upscaling program with an easy GUI. This is the easiest way to use it. Just click on add filter and add one of the two TemporalFix versions.
• Hybrid (Windows and Linux)
  Video filter toolbox with a GUI. Can be a bit overwhelming due to the amount of features and filters, but TemporalFix is one of them.
• VSGAN-tensorrt-docker (Windows and Linux)
  Command line AI upscale and interpolation toolbox that comes with both versions of TemporalFix. Rudimentary knowledge of Docker and VapourSynth is recommended, but the readme also explains it.
• mpv-cHiDeNoise-AI (Windows only)
  AI upscaling video player based on mpv, which includes the TemporalFix AI Models and a lighter modified version of TemporalFix Classic for real-time playback. Mainly intended for japanese audience.