wandarr 1.1.0

A ffmpeg transcoding workflow engine with clustered host support

wandarr - the Distributed Transcoding Toolkit

This CLI tool is a transcoding workflow manager that makes transcoding video files easier and optionally across multiple machines in parallel, using ffmpeg. It is the successor to pytranscoder and is based on the fundamental codebase of that project.

NOTE that ffmpeg version 7.0 has some wonky status output that confuses wandarr. For the time being I recommend setting your wandarr.yml "rich=" setting to "off" -- ie rich=off

Features:

• Sequential or concurrent transcoding.
• Concurrent mode allows you to make maximum use of your nVidia CUDA-enabled graphics card, Intel accelerated video (QSV), or Apple VideoToolkit.
• Preserves all streams but allows for filtering by audio and subtitle language.
• Configurable transcoding templates
• Transcode from a list of files (queue) or all on the command line
• Clustering allows use of other machines.
• On-the-fly compression monitoring and optional early job termination if not compressing as expected.

Requirements

• Linux or MacOS, Windows.
• latest ffmpeg (3.4.3-2 or higher, lower versions may still work)
• nVidia graphics card with latest nVidia CUDA drivers (optional)
• Intel CPU with QSV enabled (optional)
• MacOS VideoToolkit (optional)
• Python 3 (3.10 or higher)

Installation

pip3 install wandarr

Upgrading

Whatever method above for installing works for you, just use the --upgrade option to update, ie:

pip3 install --upgrade wandarr

Support

Please log issues or questions via the github home page for now.

Configuration

There is a supplied sample wandarr.yml config file, or you can download it from the git home. This can be customized all you like, however be sure to preserve the YAML formatting. Either specify this file on the commandline with the -y option or copy it to your home directory as .wandarr.yml (default)

In short, you will define your machine(s) used for transcoding, the transcoding capabilities of those machines, and template to tie it all together.

There are 4 sections in the file:

Section 1 - config - Global configuration information

Sample

config:
  ffmpeg:   '/opt/homebrew/bin/ffmpeg'  # path to ffmpeg for this config
  rich:     yes                         # use rich text library for nicer output

Section 2 - host definition(s)

The cluster: section is where you define all the machines in your network you intend to use for asynchronous transcoding jobs. This is optional - you need only define a single machine if that's all you are using.

Single-host sample - you can just start out like this:

cluster:
  mbp:
    os: macos
    type: local                 # the host you will run wandarr on.
    working_dir: /tmp           # best if this is an SSD
    ffmpeg: '/opt/homebrew/bin/ffmpeg'
    engines:
      - vt
    status: enabled

Multi-host Sample:

All fields in these samples are required for their respective host type and there are no optional ones. This sample cluster defines 3 hosts that will participate in a transcode job. The video files reside on a NAS.

The first host, "homeserver", is an Ubuntu machine with an NFS mount to the NAS (type=mounted). It has both a QSV-enabled Intel CPU and a discrete nVidia graphics card with CUDA support.

The second host, "winpc", is a Windows 11 machine running wandarr as an agent (type=agent). It also has an nVidia card installed but not QSV-enabled.

The final host, "mbp", is a MacBook Pro (M1 Pro) where this process is running from (type=local). It will also handle transcoding using the Apple VideoToolbox hardware acceleration feature.

cluster:
  # sample linux machine with an Intel i5 CPU/iGPU and nVidia graphics card
  homeserver:
    os: linux
    type: mounted                 # files to transcode are available via a network mount
    working_dir: /tmp             # dir to use for temp files during transcode, best to place on an SSD
    ip: 192.168.2.64              # ip or hostname of the machine
    user: marshall                # your ssh user login name. You must be able to ssh into this host w/o a password
    ffmpeg: '/usr/bin/ffmpeg'     # location of ffmpeg on the host
    engines:                      # list one of more of the video "engines" defined in the next section.  This identifies
                                  # the video handling capabilities of the host. In this sample, this host can do QSV and CUDA
      - qsv
      - cuda
      - cpu
    path-substitutions:           # for mounted types only - maps the local drive path to remote mounted one (more later)
      - '/Volumes/USB11/ /mnt/media/'
      - '/Volumes/media/ /mnt/media/'
      - '/mnt/downloads/ /mnt/downloads/'
    status: enabled               # enabled or disabled

  # sample Windows 11 machine with an nVidia graphics card
  winpc:
    os: win10
    type: agent                   # this host is running wandarr in agent mode (more later). No mounts or ssh used.
    ip: 192.168.2.61
    ffmpeg: 'c:\ffmpeg\bin\ffmpeg.exe'
    engines:
      - cuda
      - cpu
    working_dir: 'c:/temp'
    path-substitutions:
      - '/Volumes/media m:'
      - '/Volumes/USB11/media m:'
      - '/mnt/merger/media/video/ n:video\'
      - '/mnt/downloads/ z:\'
    status: enabled

  # sample MacBook Pro (M1 Pro silicon) using VideoToolkit hw acceleration
  mbp:
    os: macos
    type: local                 # the host you will run wandarr on. Not required if you will not transcode on this machine
    working_dir: /tmp
    ffmpeg: '/opt/homebrew/bin/ffmpeg'
    engines:
      - vt
      - cpu
    status: enabled

Host Types:

• local
  • The machine running wandarr. It's either your only single transcoding machine, or a member of a cluster. If you have multiple machines and will not use your "local" machine to also transcode them you need no local host defined.
• mounted
  • This machine can access the media via a network mount. Besides local, this is the next fasted option as the files can be immediately accessed. You must have an ssh password-less login to this host. You can use the ssh-copy_id tool to establish trust between machines.
• streaming
  • This machine has no network mount, so copy the file to it over first, transcode, then copy the resulting file back. This still requires ssh access like mounted.
• agent
  • This machine is running as a remote wandarr agent and requires no ssh or mounted filesystem. The tool must be installed there and started with wandarr --agent. It will use port 9567 to communicate with wandarr on your local machine to transfer files and perform transcoding. Note that this is insecure - this should only be used on your private network where you have control.

Section 3 - engines

This section defines the video transcoding capabilities of your host(s). The labels and values can be anything you like. Each section under engines defines a hardware capability. Here you see 3 - vt, qsv, and cuda representing the 3 types of hardware transcoding available to the sample hosts. Each section under quality defines as many named configurations as you need. In the sample here you see low, medium, high, and copy. These are ffmpeg options that control how your video is transformed. These are only the video options as they are the ones that can vary between hardware. While these are usable samples, you may alter then to suit your needs or add many more. Also keep in mind you do not need to use hardware acceleration, but it does make things go faster.

Sample:

engines:
  vt:         # videotoolkit hardware acceleration
    quality:
      low: "-c:v hevc_videotoolbox -qp 25 -b:v 5000K -f matroska"
      medium: "-c:v hevc_videotoolbox -preset medium -qp 23 -b:v 6000K -f matroska -max_muxing_queue_size 1024"
      high:  "-c:v hevc_videotoolbox -preset medium -qp 21 -b:v 8000K -f matroska -max_muxing_queue_size 1024"

  qsv:      # qsv hardware acceleration
    quality:
      medium: "-c:v hevc_qsv -preset medium -qp 23 -b:v 7000K -f matroska -max_muxing_queue_size 1024"
      high: "-c:v hevc_qsv -preset medium -qp 21 -b:v 7000K -f matroska -max_muxing_queue_size 1024"

  cuda:     # nvidia hardware acceleration
    quality:
      high: "-c:v hevc_nvenc -cq:v 21 -rc vbr -rc-lookahead 20 -bufsize 3M -b:v 6M -profile:v main -maxrate:v 6M -preset medium -b_ref_mode 0 -f matroska -max_muxing_queue_size 1024"
      medium: "-c:v hevc_nvenc -cq:v 23 -rc vbr -rc-lookahead 20 -bufsize 3M -b:v 7M -profile:v main -maxrate:v 7M -preset medium -b_ref_mode 0 -f matroska -max_muxing_queue_size 1024"

  cpu:      # transcode using only the CPU (ick!)
    quality:
      high-hevc: "-c:v hevc -cq:v 21 -rc vbr -rc-lookahead 20 -bufsize 3M -b:v 6M -profile:v main -maxrate:v 6M -preset medium -b_ref_mode 0 -f matroska -max_muxing_queue_size 1024"
      medium-hevc: "-c:v hevc -cq:v 23 -rc vbr -rc-lookahead 20 -bufsize 3M -b:v 7M -profile:v main -maxrate:v 7M -preset medium -b_ref_mode 0 -f matroska -max_muxing_queue_size 1024"
      high-264: "-c:v x264 -cq:v 21 -rc vbr -rc-lookahead 20 -bufsize 3M -b:v 6M -profile:v main -maxrate:v 6M -preset medium -b_ref_mode 0 -f matroska -max_muxing_queue_size 1024"
      medium-264: "-c:v x264 -cq:v 23 -rc vbr -rc-lookahead 20 -bufsize 3M -b:v 7M -profile:v main -maxrate:v 7M -preset medium -b_ref_mode 0 -f matroska -max_muxing_queue_size 1024"
      copy: "-c:v copy -f matroska"

Section 4 - templates

Templates define overall how to handle a video. As with engine definitions above, the cli section here defines the ffmpeg options for handling audio and subtitles.

In the samples below you will see examples of video-only transcoding (preserve audio as-is), full audio and video transcoding, and special case template used just to scrub out unwanted languages.

Note the video-quality element. This is used to complete the linkage from template to engine to hosts.

Sample:

templates:
  vid-only:                   # name of the template - you will use this on the commandline
    cli:                      # section for non-video ffmpeg commandline options
      audio: "-c:a copy"
      subtitles: "-c:s copy"
    video-quality: medium      # match this template to the "medium" quality defined in *engines*
    audio-lang: eng           # preserve only English audio tracks (opt).
    subtitle-lang: eng        # preserve only English subtitle tracks (opt).
    threshold: 15             # minimum required compression is %15, or terminate transcode (opt)
    threshold_check: 20       # start checking for minimum threshold at 20% (opt)
    extension: '.mkv'         # use this file extension

  vid-only-anime:
    cli:
      audio: "-c:a copy"
      subtitles: "-c:s copy"
    video-quality: medium
    audio-lang: "eng jpn"     # preserve English and Japanese audio
    subtitle-lang: eng
    threshold: 15
    threshold_check: 20
    extension: '.mkv'

  best-medium:
    cli:
      audio: "-c:a ac3 -b:a 768k"
      subtitles: "-c:s copy"
    video-quality: medium
    audio-lang: eng
    subtitle-lang: eng
    threshold: 15
    threshold_check: 20
    extension: '.mkv'

  best-medium-anime:
    cli:
      audio: "-c:a ac3 -b:a 768k"
      subtitles: "-c:s copy"
    video-quality: medium
    audio-lang: "eng jpn"
    subtitle-lang: eng
    threshold: 15
    threshold_check: 30
    extension: '.mkv'

  scrub:        # this template used only to scrub out undesired audio and subtitle tracks. no transcoding done.
    cli:
      audio: "-c:a copy"
      subtitles: "-c:s copy"
    video-quality: copy
    audio-lang: eng
    subtitle-lang: eng
    extension: '.mkv'

  scrub-anime:  # this template used only to scrub out undesired audio and subtitle tracks. no transcoding done.
    cli:
      audio: "-c:a copy"
      subtitles: "-c:s copy"
    video-quality: copy
    audio-lang: "eng jpn"
    subtitle-lang: eng
    extension: '.mkv'

Putting it all together

Here's how to read the samples above in their entirety.

In the vid-only template above, we defined the ffmpeg options to handle audio and subtitles. In this case, we're just copying with no changes. Now we need to know the ffmpeg options for handling video. The video-quality value of medium matches all medium definitions in engines. This tells wandarr that any hosts that match an engine definition containing a medium will be a eligible match for this template. So for medium we have told wandarr how to transcode video for video toolbox, intel qsv, and nvidia cuda. Whichever hosts are associated to those engines may be selected to do the job.

So, a template relates to an engine quality. A quality defines ffmpeg options for all supported hardware to accomplish the same result. A host relates to one or more engines. This is how a host is selected for transcoding, and how to do it.

You can be as basic or complex as you need. The typical user only needs 2 or 3 templates.

Running

The default behavior is to remove the original video file after encoding and replace it with the new version. If you want to keep the source be sure to use the -k parameter. The work file will be placed in the same folder as the source with the same name and a .tmp extension while being encoded.

usage: main.py [-h] [-v] [-i] [-k] [--dry-run] [-y CONFIGFILE_NAME] [--agent] [-t TEMPLATE] [--hosts HOST_OVERRIDE] [--from-file FROM_FILE] [filename ...]

wandarr (ver 1.0.0)

positional arguments:
  filename

options:
  -h, --help            show this help message and exit
  -v                    verbose mode
  -i                    show technical info on files and stop
  -k                    keep source (do not replace)
  --dry-run             Test run, show steps but don't change anything
  -y CONFIGFILE_NAME    Full path to configuration file. Default is ~/.wandarr.yml
  --agent               Start in agent mode on a host and listen for transcode requests from other wandarr.
  -t TEMPLATE           Template name to use for transcode jobs
  --hosts HOST_OVERRIDE
                        Only run transcode on given host(s), comma-separated
  --from-file FROM_FILE
                        Filename that contains list of full paths of files to transcode

Examples:

To get help and version number:

   wandarr -h

Show me the technical goods on some files:

   wandarr -i *.mkv

To transcode 2 files using a specific template:

    wandarr -t my_fave_x264 /tmp/video1.mp4 /tmp/video2.mp4

To auto transcode everything listed in a specific file:

    wandarr -t tv --from-file /tmp/queue.txt

To do a test run without transcoding:

    wandarr -t movie-high --dry-run atestvideo.mp4

To use a specific yml file:

    wandarr -y /home/me/etc/quickstart.yml -t tv *.mp4

To transcode on a specific host only:

    wandarr -t tv --host workstation *.mp4