Deployments

YTSync

YTSync is a lightweight API, equipped with Telegram Bot to download a playlist and asynchronously transfer it to a remote server via rsync (ssh)

Features

• Supports both webhooks and long-polling
• Users can switch between webhooks and long-polling through API
• Sequential downloads (to prevent requests being throttled/blocked)
• Cold start (delayed) for the first download with a custom cooldown interval
• Built-in retry mechanism with an exponential back off factor for remote transfers
• Automatically deletes [Optional] local files after transferring to a remote server
• Built-in support to check file presence on the remote server before requesting download/file-transfer

Environment variables

Server Settings

• host: Hostname to run API server. Defaults to 0.0.0.0 [OR] localhost
• port: Port number to run the API server. Defaults to 4483
• tz: IANA time zone identifier. Defaults to server's local timezone
• log_config: Dict config or filepath for log configuration. Defaults to logging.basicConfig

Profile Settings

• name: Name of the profile.
• apikey: Key to access via API. Required for API access
• bot_chat_id: List of bot ids to allow. Required for telegram access
• bot_username: List of bot usernames to allow. Required for telegram access

Telegram Settings

• bot_token: Telegram bot token. Required for telegram access
• poll_interval: Number of seconds between each request to poll. Defaults to 2
• bot_webhook: Telegram bot webhook URL. [Optional]
• bot_webhook_ip: Webhook IP address. [Optional]
• bot_endpoint: API endpoint to serve the webhook. Defaults to /telegram-webhook
• bot_secret: Secret key to verify webhook requests. [Optional]
• bot_certificate: Certificate filepath for webhook server (in case of self-signed certificate) [Optional]

yt-dlp Settings [Optional]

• download_tester: Boolean flag to skip download, and create placeholder files for testing. Defaults to False
• cookie_file: Path to the cookie file.
• source_address: IP address for the requesting source.
• proxy_url: URL of the proxy server.

FileIO Settings

• data_dir: Directory to store the database. Defaults to data
• logs_dir: Directory to store logs. Defaults to logs
• audio_dir: Directory to store audio files. Defaults to audio
• video_dir: Directory to store video files. Defaults to video

Remote machines can't choose, audio and video directory. It is auto-created with default values.

Concurrency & Tolerance Settings

• max_transfers: Maximum number of concurrent transfers to perform. Defaults to the number of CPU cores
• max_retries: Maximum number of retries for rsync and telegram polling. Defaults to 10
• max_timeout: Maximum number of seconds to wait before timing out the CLI download and rsync transfer. Defaults to 60
• backoff_factor: Back off factor between each retry attempt. Defaults to 3
• max_error_threshold: Percentage of individual URLs to verify before downloading the entire playlist. Defaults to 30
• response_timeout: Maximum number of seconds to wait before timing out the client request. Defaults to 30

Queue Settings

• delayed_start: Perform a cold start (delayed) for the first download. Defaults to False
• next_buffer: Number of seconds to simulate time taken for a download. Defaults to 60
• cooldown_interval: Number of seconds to wait before processing next in queue. Defaults to 300

Analytics Settings [Optional]

• checkpoint_retention: Number of days/weeks/months to retain the checkpoint data. Defaults to 3d

Remote Settings [Optional]

• remote_host: Hostname [OR] IP address of the remote server.
• remote_user: Username to connect to the remote server.
• remote_path: Directory path on the remote server, to transfer downloaded files to.
• delete_after_sync: Boolean flag to delete local files after transferring to the remote server.

Limitations

• Environment variables can be read from a .env or .json file, but docker context only supports .env files.
• Profiles will be created by default within the chosen audio_dir and video_dir directories.
• YTSync does not support threaded downloads, and will download sequentially to avoid being throttled/blocked by the source server.
• YTSync runs on a single process, and will not spawn threads. However, downloads/transfers are asynchronous; transfers are submitted immediately after a download is complete.

SSH setup

Allow remote server connection without requiring a password [OR] private key during run-time

ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519
ssh-copy-id user@receiver_ip
ssh user@receiver_ip

Docker

Mount the local ~/.ssh volume to allow reading known hosts

docker run \
  -v ~/.ssh/id_ed25519:/root/.ssh/id_ed25519:ro \
  -v ~/.ssh/known_hosts:/root/.ssh/known_hosts:ro \
  ytsync

Linting

pre-commit will ensure linting

Requirement

pip install pre-commit

Usage

pre-commit run --all-files

Pypi Package

https://pypi.org/project/YTSync/

License & copyright

© Vignesh Rao

Licensed under the MIT License