Download videos to a remote server using youtube-dl via a REST API
Project description
ytdl-server
ytdl-server is a program that allows you to download videos onto a remote server using youtube-dl.
Using youtube-dl to download videos onto a network file share is inefficient since the video file has to pass through your workstation. Additionally, many of the post-processing options that youtube-dl provides will create a temporary copy of the file, which means that the amount of data uploaded over the network can be 3-4x the size of the actual file.
ytdl-server solves this by allowing you to download videos directly to the remote server via a REST API.
ytdl-server also supports high-availability, which means that you can perform maintenance without downtime.
[TOC]
How it works
ytdl-server has two main components: the Flask REST API and the Celery worker.
The REST API communicates with the user in order to create jobs. The jobs are then run by the worker in order to download the desired videos.
The complete structure of ytdl-server is shown in the below diagram:
Included in ytdl-server means that the component is part of ytdl-server.
Included in the Docker images means that the component is built into the Docker images. You'll need to install it separately if you're not using them.
Not included means that the component is not part of ytdl-server and needs to be installed separately.
-
Reverse proxy / Load balancer (not included)
Used to forward user-requests to the WSGI server(s).
Some available options are NGINX and Caddy.
Load balancing is only needed if you're running multiple WSGI server instances.
-
WSGI server (included in the Docker images)
Used to run the Flask REST API(s).
Some available options are Gunicorn and gevent.
If you're using the Docker images, Gunicorn is included.
If you're running multiple REST API instances, each one needs its own WSGI server instance.
-
Flask REST API (included in ytdl-server)
Used to create and manage jobs.
You can run multiple instances if you want high-availability. Otherwise, a single instance is fine.
-
Celery broker (not included)
Used to dispatch jobs created by the REST API to a Celery worker.
You can use any broker supported by Celery, including RabbitMQ and Redis.
-
Celery worker (included in ytdl-server)
Used to run jobs (download videos).
You can run multiple instances if you want high-availability. Otherwise, a single instance is fine.
-
PostgreSQL database (not included)
Used to store information about jobs, and also functions as the Celery result backend.
Installation
Docker (recommended)
Prebuilt Docker images are provided. See Registry for a list of supported tags.
An example Docker Compose file is provided here. To use it, copy the Compose file to a directory, and run the following command to start it:
docker-compose up
The REST API will be accessible at port 8000. See Usage for how to use it.
If you want to use a custom config file, overwrite the existing config file at
/config/ytdl-server.yml
. See Configuration for available
options.
Manual
ytdl-server requires Python 3.9+.
Install from PyPI:
pip3 install 'ytdl-server[dbapi]'
Install from source:
git clone 'https://gitlab.com/adralioh/ytdl-server.git'
pip3 install './ytdl-server[dbapi]'
The
[dbapi]
extra installs Psycopg2, which is used to connect to the PostgreSQL database. You must install a DBAPI for ytdl-server to work. You can alternatively installpsycopg2-binary
if you don't want to compile it from source. See Installation - Psycopg 2 for details.
If you're using Redis or Amazon SQS as the broker, you must also install the following extra dependencies:
# Redis
pip3 install 'celery[redis]'
# Amazon SQS
pip3 install 'celery[sqs]'
Run the REST API using a WSGI server such as Gunicorn:
gunicorn -w 4 'ytdl_server.flask:with_logging()'
Run the Celery worker using Celery:
celery -A 'ytdl_server.run_celery' worker
You also need to set up the reverse proxy, Celery broker, and PostgreSQL database. See How it works for information, and see Configuration for how to configure ytdl-server to use the database and broker.
Updating
When updating ytdl-server, refer to the versioning scheme and the changelog for information about what has changed.
Configuration
See Configuration.
Usage
Videos are downloaded via jobs. You create a new job whenever you want to download one or more videos.
The following examples access the ytdl-server directly using curl. See Frontends for more user-friendly ways of accessing it.
Create a job:
curl ytdl.example.com/jobs/ \
-H 'Content-Type: application/json' \
-d '{
"urls": [ "https://www.youtube.com/watch?v=AAAAAAAAAAA",
"https://www.youtube.com/watch?v=BBBBBBBBBBB" ],
"ytdl_opts": { "format": "best",
"outtmpl": "%(id)s.%(ext)s" }
}'
Jobs are created asynchronously, which means that the videos will be downloaded in the background.
You can check the status of a job as follows:
curl ytdl.example.com/jobs/ffedda2d-7fa4-4839-a705-88c893e3f942
ffedda2d-7fa4-4839-a705-88c893e3f942 is the job ID. This is obtained from the output of the previous command where you created the job.
You can also cancel a running job:
curl ytdl.example.com/jobs/ffedda2d-7fa4-4839-a705-88c893e3f942 \
-X PATCH \
-H 'Content-Type: application/json' \
-d '{
"status": "cancelled"
}'
For complete documentation about the REST API, see REST API.
Registry
Prebuit Docker images are provided at registry.gitlab.com/adralioh/ytdl-server
for the REST API and the Celery worker.
The supported tags are rebuilt monthly in order to provide the latest security updates.
REST API
Images for the REST API are available at
registry.gitlab.com/adralioh/ytdl-server/api
.
The images include Gunicorn, which is used as the WSGI server.
Supported tags:
latest
,1
,1.1
,1.1.8
Celery worker
Images for the Celery worker are available at
registry.gitlab.com/adralioh/ytdl-server/worker
.
The yt_dlp tags use yt-dlp instead of
youtube-dl. You also need to set the YTDL_MODULE env-var to yt_dlp
on the
REST API when using these.
The ffmpeg tags include FFmpeg, which is required for many of youtube-dl's post-processing options.
Supported tags:
latest
,1
,1.1
,1.1.8
yt_dlp
,1-yt_dlp
,1.1-yt_dlp
,1.1.8-yt_dlp
ffmpeg
,1-ffmpeg
,1.1-ffmpeg
,1.1.8-ffmpeg
yt_dlp-ffmpeg
,1-yt_dlp-ffmpeg
,1.1-yt_dlp-ffmpeg
,1.1.8-yt_dlp-ffmpeg
Testing and development
See tests
Frontends
- ytcl - Command-line interface with syntax similar to youtube-dl
Frequently asked questions
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file ytdl-server-1.1.8.tar.gz
.
File metadata
- Download URL: ytdl-server-1.1.8.tar.gz
- Upload date:
- Size: 53.6 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | e326b4257b6cd97ff59651831db1c8c5f5cef942f0c72e960c455409bb15f831 |
|
MD5 | 97c629ee55d34fc24462dd2585d0d11e |
|
BLAKE2b-256 | 4d3edc5753b1ccf4c97b4f3ce25680b5b7ce7b5684087a6f87d7a8eb37bc9361 |
File details
Details for the file ytdl_server-1.1.8-py3-none-any.whl
.
File metadata
- Download URL: ytdl_server-1.1.8-py3-none-any.whl
- Upload date:
- Size: 59.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.2 CPython/3.9.18
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8182fa5040e1470cc8ad6cbad9b1e07816441ea8393d0ee2ba5a85dc8cb3c70a |
|
MD5 | 3eb141e3543574f2a3705acb192a6624 |
|
BLAKE2b-256 | ad6b98f6d2af7dfa0ebd2bd0ccdcf7f70db840e1513ae301d068f7aec70163cb |