Skip to main content

Download videos to a remote server using youtube-dl via a REST API

Project description

ytdl-server

ytdl-server is a program that allows you to download videos onto a remote server using youtube-dl.

Using youtube-dl to download videos onto a network file share is inefficient since the video file has to pass through your workstation. Additionally, many of the post-processing options that youtube-dl provides will create a temporary copy of the file, which means that the amount of data uploaded over the network can be 3-4x the size of the actual file.

ytdl-server solves this by allowing you to download videos directly to the remote server via a REST API.

ytdl-server also supports high-availability, which means that you can perform maintenance without downtime.

[TOC]

How it works

ytdl-server has two main components: the Flask REST API and the Celery worker.

The REST API communicates with the user in order to create jobs. The jobs are then run by the worker in order to download the desired videos.

The complete structure of ytdl-server is shown in the below diagram:

Diagram of ytdl-server's structure

Included in ytdl-server means that the component is part of ytdl-server.

Included in the Docker images means that the component is built into the Docker images. You'll need to install it separately if you're not using them.

Not included means that the component is not part of ytdl-server and needs to be installed separately.

  • Reverse proxy / Load balancer (not included)

    Used to forward user-requests to the WSGI server(s).

    Some available options are NGINX and Caddy.

    Load balancing is only needed if you're running multiple WSGI server instances.

  • WSGI server (included in the Docker images)

    Used to run the Flask REST API(s).

    Some available options are Gunicorn and gevent.

    If you're using the Docker images, Gunicorn is included.

    If you're running multiple REST API instances, each one needs its own WSGI server instance.

  • Flask REST API (included in ytdl-server)

    Used to create and manage jobs.

    You can run multiple instances if you want high-availability. Otherwise, a single instance is fine.

  • Celery broker (not included)

    Used to dispatch jobs created by the REST API to a Celery worker.

    You can use any broker supported by Celery, including RabbitMQ and Redis.

  • Celery worker (included in ytdl-server)

    Used to run jobs (download videos).

    You can run multiple instances if you want high-availability. Otherwise, a single instance is fine.

  • PostgreSQL database (not included)

    Used to store information about jobs, and also functions as the Celery result backend.

Installation

Docker (recommended)

Prebuilt Docker images are provided. See Registry for a list of supported tags.

An example Docker Compose file is provided here. To use it, copy the Compose file to a directory, and run the following command to start it:

docker-compose up

The REST API will be accessible at port 8000. See Usage for how to use it.

If you want to use a custom config file, overwrite the existing config file at /config/ytdl-server.yml. See Configuration for available options.

Manual

ytdl-server requires Python 3.9+.

Install from PyPI:

pip3 install 'ytdl-server[dbapi]'

Install from source:

git clone 'https://gitlab.com/adralioh/ytdl-server.git'
pip3 install './ytdl-server[dbapi]'

The [dbapi] extra installs Psycopg2, which is used to connect to the PostgreSQL database. You must install a DBAPI for ytdl-server to work. You can alternatively install psycopg2-binary if you don't want to compile it from source. See Installation - Psycopg 2 for details.

If you're using Redis or Amazon SQS as the broker, you must also install the following extra dependencies:

# Redis
pip3 install 'celery[redis]'
# Amazon SQS
pip3 install 'celery[sqs]'

Run the REST API using a WSGI server such as Gunicorn:

gunicorn -w 4 'ytdl_server.flask:with_logging()'

Run the Celery worker using Celery:

celery -A 'ytdl_server.run_celery' worker

You also need to set up the reverse proxy, Celery broker, and PostgreSQL database. See How it works for information, and see Configuration for how to configure ytdl-server to use the database and broker.

Updating

When updating ytdl-server, refer to the versioning scheme and the changelog for information about what has changed.

Configuration

See Configuration.

Usage

Videos are downloaded via jobs. You create a new job whenever you want to download one or more videos.

The following examples access the ytdl-server directly using curl. See Frontends for more user-friendly ways of accessing it.

Create a job:

curl ytdl.example.com/jobs/ \
     -H 'Content-Type: application/json' \
     -d '{
           "urls": [ "https://www.youtube.com/watch?v=AAAAAAAAAAA",
                     "https://www.youtube.com/watch?v=BBBBBBBBBBB" ],
           "ytdl_opts": { "format": "best",
                          "outtmpl": "%(id)s.%(ext)s" }
         }'

Jobs are created asynchronously, which means that the videos will be downloaded in the background.

You can check the status of a job as follows:

curl ytdl.example.com/jobs/ffedda2d-7fa4-4839-a705-88c893e3f942

ffedda2d-7fa4-4839-a705-88c893e3f942 is the job ID. This is obtained from the output of the previous command where you created the job.

You can also cancel a running job:

curl ytdl.example.com/jobs/ffedda2d-7fa4-4839-a705-88c893e3f942 \
     -X PATCH \
     -H 'Content-Type: application/json' \
     -d '{
           "status": "cancelled"
         }'

For complete documentation about the REST API, see REST API.

Registry

Prebuit Docker images are provided at registry.gitlab.com/adralioh/ytdl-server for the REST API and the Celery worker.

The supported tags are rebuilt monthly in order to provide the latest security updates.

REST API

Images for the REST API are available at registry.gitlab.com/adralioh/ytdl-server/api.

The images include Gunicorn, which is used as the WSGI server.

Supported tags:

  • latest, 1, 1.1, 1.1.8

Celery worker

Images for the Celery worker are available at registry.gitlab.com/adralioh/ytdl-server/worker.

The yt_dlp tags use yt-dlp instead of youtube-dl. You also need to set the YTDL_MODULE env-var to yt_dlp on the REST API when using these.

The ffmpeg tags include FFmpeg, which is required for many of youtube-dl's post-processing options.

Supported tags:

  • latest, 1, 1.1, 1.1.8
  • yt_dlp, 1-yt_dlp, 1.1-yt_dlp, 1.1.8-yt_dlp
  • ffmpeg, 1-ffmpeg, 1.1-ffmpeg, 1.1.8-ffmpeg
  • yt_dlp-ffmpeg, 1-yt_dlp-ffmpeg, 1.1-yt_dlp-ffmpeg, 1.1.8-yt_dlp-ffmpeg

Testing and development

See tests

Frontends

  • ytcl - Command-line interface with syntax similar to youtube-dl

Frequently asked questions

See Frequently asked questions.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ytdl-server-1.1.8.tar.gz (53.6 kB view details)

Uploaded Source

Built Distribution

ytdl_server-1.1.8-py3-none-any.whl (59.5 kB view details)

Uploaded Python 3

File details

Details for the file ytdl-server-1.1.8.tar.gz.

File metadata

  • Download URL: ytdl-server-1.1.8.tar.gz
  • Upload date:
  • Size: 53.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.18

File hashes

Hashes for ytdl-server-1.1.8.tar.gz
Algorithm Hash digest
SHA256 e326b4257b6cd97ff59651831db1c8c5f5cef942f0c72e960c455409bb15f831
MD5 97c629ee55d34fc24462dd2585d0d11e
BLAKE2b-256 4d3edc5753b1ccf4c97b4f3ce25680b5b7ce7b5684087a6f87d7a8eb37bc9361

See more details on using hashes here.

File details

Details for the file ytdl_server-1.1.8-py3-none-any.whl.

File metadata

  • Download URL: ytdl_server-1.1.8-py3-none-any.whl
  • Upload date:
  • Size: 59.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.18

File hashes

Hashes for ytdl_server-1.1.8-py3-none-any.whl
Algorithm Hash digest
SHA256 8182fa5040e1470cc8ad6cbad9b1e07816441ea8393d0ee2ba5a85dc8cb3c70a
MD5 3eb141e3543574f2a3705acb192a6624
BLAKE2b-256 ad6b98f6d2af7dfa0ebd2bd0ccdcf7f70db840e1513ae301d068f7aec70163cb

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page