ls-dlp 0.0.5

Garbage youtube livestream downloader

ls-dlp

This is a forked project of livestream_dl.

A robust YouTube livestream downloader that combines the reliability of yt-dlp with the fragment-based downloading principles of ytarchive.

This tool focuses on using yt-dlp for stream information extraction, ensuring compatibility with YouTube's changing infrastructure. It is designed to handle live recording with the ability to recover streams if they become unavailable (e.g., go private) during the download process.

Key Features

• Robust Downloading: Uses yt-dlp for extraction, minimizing breakage from YouTube updates.
• Stream Recovery: capable of recovering streams that go private during recording (requires specific configuration).
• Dual Write Modes: Supports both direct .ts file writing (low disk IO) and SQLite-backed downloading (safer for unstable connections).
• Live Chat: Integrated support for downloading live chat (via yt-dlp) and bundling it into a ZIP file.
• Channel Monitoring: Automated monitoring of channels for upcoming or live streams.
• Protocol Fallbacks: Optional support for DASH and HLS (m3u8) protocols to avoid modifying yt-dlp source code.

---

Requirements

See Dockerfile

Installation

pip install ls-dlp

Usage

ls-dlp [OPTIONS] [VIDEO_URL_OR_ID]

Examples

Basic download:

ls-dlp \
  --threads 4 \
  --dash \
  --m3u8 \
  --write-thumbnail \
  --embed-thumbnail \
  --wait-for-video "60:600" \
  --clean-info-json \
  --remove-ip-from-json \
  --live-chat \
  --resolution "best" \
  --write-info-json \
  --log-level "INFO" -- "[VIDEO_ID]"

Download with live chat and embed thumbnail:

ls-dlp --live-chat --threads 4 --embed-thumbnail VIDEO_ID

Monitor a channel for streams:

ls-dlp --monitor-channel --threads 4 --dash --m3u8 --wait-for-video 60 CHANNEL_ID

---

Command Line Options

General Configuration

Option	Default	Description
ID	None	The video URL or ID (Positional argument).
--help, -h	N/A	Show the help message and exit.
--cookies	None	Path to a Netscape-formatted cookies file (required for age-gated or members-only content).
--temp-folder	None	Path for temporary files (database, segments). Supports yt-dlp output formatting.
--output	%(fulltitle)s (%(id)s)	Path/filename for output files. Supports yt-dlp output formatting. A good example is: "[%(release_date,upload_date)s] %(fulltitle).120B [%(channel)s] (%(id)s)"
--ext	None	Force the extension of the final video file (e.g., .mp4, .mkv).
--json-file	None	Path to an existing yt-dlp info.json file. Overrides ID and skips retrieving URLs from the web.
--write-ffmpeg-command	False	Writes the final FFmpeg command used to merge the file to a .txt file for debugging or manual merging.

Download & Formatting

Option	Default	Description
--resolution	best	Desired resolution. Can be best, audio_only, or a custom yt-dlp format filter. -Audio is always set to "ba" (best audio) regardless of filters. -"best" is converted to "bv". -If unspecified, a prompt will appear.
--custom-sort	None	Custom sorting algorithm for formats based on yt-dlp sorting syntax.
--threads	1	Number of download threads per format. Total threads = this value × 2 (for video+audio).
--batch-size	5	Number of segments downloaded before the temporary database is committed to disk (reduces disk IO).
--segment-retries	10	Number of times to retry downloading a specific segment before failing.
--dash	False	Use DASH URLs as fallback. Does not require yt-dlp modification but prevents stream recovery if DASH URLs are used.
--m3u8	False	Use HLS (m3u8) URLs as fallback. Combined audio/video streams (halves requests). Does not require yt-dlp modification but prevents recovery if m3u8 URLs are used.
--force-m3u8	False	Forces the use of m3u8 stream URLs.
--wait-for-video	None	Wait time (int) or range (min:max) to wait for a video to start or become available. Refer to yt-dlp's documentation
--ytdlp-options	None	JSON string of additional yt-dlp options (overwrites conflicts). E.g '{"extractor_args":{"youtubepot-bgutilhttp":{"base_url":["http://bgutil-provider:4416"]}}}' to use Potoken retrieval. Refer to full options for more examples.

Processing & Output

Option	Default	Description
--no-merge	False	Do not merge video and audio using FFmpeg after download.
--merge	False	Force merging video using FFmpeg (overrides --no-merge).
--write-thumbnail	False	Save the video thumbnail to a separate file.
--embed-thumbnail	False	Embed the thumbnail into the final video file (ignored if --no-merge is active).
--write-info-json	False	Write the video metadata to an info.json file.
--write-description	False	Write the video description to a separate text file.
--keep-temp-files	False	Keep all temporary files (database and TS files) after finishing.
--keep-ts-files	False	Keep the intermediate TS files but delete the database.
--live-chat	False	Download live chat (requires yt-dlp).
--stop-chat-when-done	300	Max seconds to wait for chat download to finish after stream ends.

Database & Storage Modes

Option	Default	Description
--database-in-memory	False	Keep the segment database in RAM. High memory usage; not recommended for long streams.
--direct-to-ts	False	Write directly to a .ts file instead of using a SQLite database. See "Downloader methods" below.
--keep-database-file	False	Keep the database file after completion. If using --direct-to-ts, keeps the state file.

Recovery & Resilience

Option	Default	Description
--recovery	False	Puts the downloader directly into stream recovery mode.
--force-recover-merge	False	Forces merging to the final file even if not all segments were successfully recovered.
--recovery-failure-tolerance	0	Max number of fragments allowed to fail during recovery without throwing a final error.
--wait-limit	0	Max wait intervals (~10s each) for new segments. If 0, waits until status changes to post_live or was_live.

Network & Logging

Option	Default	Description
--proxy	None	Proxy URL (string) or JSON string for multiple methods. First proxy used for yt-dlp/chat.
--ipv4	False	Force IPv4 only.
--ipv6	False	Force IPv6 only.
--log-level	INFO	Logging level: DEBUG, VERBOSE, INFO, WARNING, ERROR, CRITICAL. Verbose logging is a custom level that includes the INFO logs of yt-dlp.
--no-console	False	Disable printing log messages to the console.
--log-file	None	Path to a file where log messages will be saved.
--stats-as-json	False	Prints download statistics as a JSON string (bypasses log level).
--new-line	False	Ensures console messages always print to a new line (useful for some terminals).

Channel Monitoring

Option	Default	Description
--monitor-channel	False	Enables channel monitoring mode. Use the Channel ID in the ID argument.
--members-only	False	Monitor the 'Members Only' playlist instead of public streams. Requires cookies.
--upcoming-lookahead	24	Maximum time (in hours) to look ahead for upcoming videos to schedule.
--playlist-items	50	Maximum number of playlist items to check when monitoring.

Metadata Privacy

Option	Default	Description
--remove-ip-from-json	False	Replaces IP addresses in info.json with 0.0.0.0.
--clean-urls	False	Removes potentially identifiable stream URLs from info.json.
--clean-info-json	False	Enables yt-dlp's internal clean-info-json option.

Full options

$ ls-dlp --help
usage: runner.py [-h] [--resolution RESOLUTION] [--custom-sort CUSTOM_SORT] [--threads THREADS] [--batch-size BATCH_SIZE] [--segment-retries SEGMENT_RETRIES] [--no-merge] [--merge] [--cookies COOKIES] [--output OUTPUT] [--ext EXT] [--temp-folder TEMP_FOLDER] [--write-thumbnail] [--embed-thumbnail]      
                 [--write-info-json] [--write-description] [--keep-temp-files] [--keep-ts-files] [--live-chat] [--keep-database-file] [--recovery] [--force-recover-merge] [--recovery-failure-tolerance RECOVERY_FAILURE_TOLERANCE] [--wait-limit WAIT_LIMIT] [--database-in-memory] [--direct-to-ts]
                 [--wait-for-video WAIT_FOR_VIDEO] [--json-file JSON_FILE] [--remove-ip-from-json] [--clean-urls] [--clean-info-json] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [--no-console] [--log-file LOG_FILE] [--write-ffmpeg-command] [--stats-as-json] [--ytdlp-options YTDLP_OPTIONS] [--dash]
                 [--m3u8] [--force-m3u8] [--proxy [PROXY]] [--ipv4 | --ipv6] [--stop-chat-when-done STOP_CHAT_WHEN_DONE] [--new-line] [--monitor-channel] [--members-only] [--upcoming-lookahead UPCOMING_LOOKAHEAD] [--playlist-items PLAYLIST_ITEMS]
                 [ID]

Download YouTube livestream (https://github.com/CanOfSocks/livestream_dl)

positional arguments:
  ID                    The video URL or ID

options:
  -h, --help            show this help message and exit
  --resolution RESOLUTION
                        Desired resolution. Can be best, audio_only or a custom filter based off yt-dlp's format filtering: https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#filtering-formats. Audio will always be set as "ba" (best audio) regardless of filters set. "best" will be converted to "bv" A prompt     
                        will be displayed if no value is entered
  --custom-sort CUSTOM_SORT
                        Custom sorting algorithm for formats based off yt-dlp's format sorting: https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#sorting-formats
  --threads THREADS     Number of download threads per format. This will be 2x for an video and audio download. Default: 1
  --batch-size BATCH_SIZE
                        Number of segments before the temporary database is committed to disk. This is useful for reducing disk access instances. Default: 5
  --segment-retries SEGMENT_RETRIES
                        Number of times to retry grabbing a segment. Default: 10
  --no-merge            Don't merge video using ffmpeg
  --merge               Merge video using ffmpeg, overrides --no-merge
  --cookies COOKIES     Path to cookies file
  --output OUTPUT       Path/file name for output files. Supports yt-dlp output formatting
  --ext EXT             Force extension of video file. E.g. '.mp4'
  --temp-folder TEMP_FOLDER
                        Path for temporary files. Supports yt-dlp output formatting
  --write-thumbnail     Write thumbnail to file
  --embed-thumbnail     Embed thumbnail into final file. Ignored if --no-merge is used
  --write-info-json     Write info.json to file
  --write-description   Write description to file
  --keep-temp-files     Keep all temp files i.e. database and/or ts files
  --keep-ts-files       Keep all ts files
  --live-chat           Get Live chat
  --keep-database-file  Keep database file. If using with --direct-to-ts, this keeps the state file
  --recovery            Puts downloader into stream recovery mode
  --force-recover-merge
                        Forces merging to final file even if all segments could not be recovered
  --recovery-failure-tolerance RECOVERY_FAILURE_TOLERANCE
                        Maximum number of fragments that fail to download (exceed the retry limit) and not throw an error. May cause unexpected issues when merging to .ts file and remuxing. Default: 0
  --wait-limit WAIT_LIMIT
                        Set maximum number of wait intervals for new segments. Each wait interval is ~10s (e.g. a value of 20 would be 200s). A minimum of value of 20 is recommended. Stream URLs are refreshed every 10 intervals. A value of 0 wait until the video moves into 'was_live' or 'post_live' status.       
                        Default: 0
  --database-in-memory  Keep stream segments database in memory. Requires a lot of RAM (Not recommended)
  --direct-to-ts        Write directly to ts file instead of database. May use more RAM if a segment is slow to download. This overwrites most database options
  --wait-for-video WAIT_FOR_VIDEO
                        Wait time (int) or Minimum and maximum (min:max) interval to wait for a video
  --json-file JSON_FILE
                        Path to existing yt-dlp info.json file. Overrides ID and skips retrieving URLs
  --remove-ip-from-json
                        Replaces IP entries in info.json with 0.0.0.0
  --clean-urls          Removes stream URLs from info.json that contain potentially identifiable information. These URLs are usually useless once they have expired
  --clean-info-json     Enables yt-dlp's 'clean-info-json' option
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Set the logging level. Default is INFO.
  --no-console          Do not log messages to the console.
  --log-file LOG_FILE   Path to the log file where messages will be saved.
  --write-ffmpeg-command
                        Writes FFmpeg command to a txt file
  --stats-as-json       Prints stats as a JSON formatted string. Bypasses logging and prints regardless of log level
  --ytdlp-options YTDLP_OPTIONS
                        Additional yt-dlp options as a JSON string. Overwrites any options that are already defined by other options. Available options: https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/YoutubeDL.py#L183. E.g. '{"extractor_args": {"youtube": {"player_client": ["web_creator"]}, "youtubepot-     
                        bgutilhttp":{ "base_url": ["http://10.1.1.40:4416"]}}}' if you have installed the potoken plugin
  --dash                Gets any available DASH urls as a fallback to adaptive URLs. Dash URLs do not require yt-dlp modification to be used, but can't be used for stream recovery and can cause large info.json files when a stream is in the 'post_live' status
  --m3u8                Gets any available m3u8 urls as a fallback to adaptive URLs. m3u8 URLs do not require yt-dlp modification to be used, but can't be used for stream recovery. m3u8 URLs provide both video and audio in each fragment and could allow for the amount of segment download requests to be halved     
  --force-m3u8          Forces use of m3u8 stream URLs
  --proxy [PROXY]       (Requires testing) Specify proxy to use for web requests. Can be a string for a single proxy or a JSON formatted string to specify multiple methods. For multiple, refer to format https://requests.readthedocs.io/en/latest/user/advanced/#proxies. The first proxy specified will be used for   
                        yt-dlp and live chat functions.
  --ipv4                Force IPv4 only
  --ipv6                Force IPv6 only
  --stop-chat-when-done STOP_CHAT_WHEN_DONE
                        Wait a maximum of X seconds after a stream is finished to download live chat. Default: 300. This is useful if waiting for chat to end causes hanging.
  --new-line            Console messages always print to new line. (Currently only ensured for stats output)
  --monitor-channel     Use monitor channel feature (Alpha). Specify channel ID in 'ID' argument
  --members-only        Monitor 'Members Only' playlist for streams instead of 'Streams' playlist. Requires cookies.
  --upcoming-lookahead UPCOMING_LOOKAHEAD
                        Maximum time (in hours) to start a downloader instance for a video. Default: 24
  --playlist-items PLAYLIST_ITEMS
                        Maximum number of playlist items to check. Default: 50

---

Downloader methods

There are two download methods available for livestream downloading, using a SQLite database and writing directly to a ts file. For stream recovery, only the SQLite option is available. Merging to a video or audio file (.mp4, .mkv, .ogg etc) will require an extra write of information. The default method is the SQLite method.

Direct writing to a .ts file

Writing directly to a ts file helps reduce the number of writes to a disk during the recording as it is only necessary to write to the final ts file. This may reduce disk wear over a long period of time of downloading many streams. As segments are downloaded, the sequence number of the segment is used to decide which segment will be appended to the ts file next. If the "next" segment is not available to be written to the disk, other segments will be stored in RAM until the respective previous segment has been written to the disk. This performs in a similar way to ytarchive. To reduce file opening and closing actions on the file system, if multiple sequential segments can be written at once, they will be written into the file at once. A state file is saved each time segments are written to the disk, saving the latest segment and size of the file, so the downloader doesn't need to re-download segments should the downloading session stop for any reason. Segment downloads are not guaranteed to finish in a sequential order if more than 1 thread is used. If a segment is particularly slow at downloading, this may increase ram usage significantly while subsequent segments are saved in RAM.

For regular livestream recording that doesn't experience frequent segment download errors, using the direct writing method will work well.

SQLite

The SQLite downloader method is used to improve handling of non-sequential successful segment downloads. This works by creating and using a basic table of the segment number as an ID and a blob to store the segment data. Once all segments are downloaded, a query is executed to sort all of the downloaded segments into the correct order and saved to a .ts file. This helps significantly to manage downloaded segments when failures occur often, such as unavailable stream recovery. By writing to the database and a final .ts file, this will require 2 full writes of the downloaded data, which may increase wear on flash-based systems. For most users, this increased wear will not be significant in the long-term. This improves over saving individual segment files like ytarchive-raw-go as all the downloaded segments are encapsulated into a single file, making it easier for the file system to handle. In the SQLite method, the existence of a downloaded segment is checked before it is downloaded. This allows failed segments to be "looped back to" later without causing other slowdowns, and ensures some information is saved for a segment (even if it is empty, as is the case sometimes).

---

Known Issues

• Concurrent Futures: The downloader uses concurrent.futures for thread management. While robust, stopping the downloader gracefully (e.g., via Keyboard Interrupt) can sometimes be delayed if a thread is stuck on a request.
• Stream Recovery: This feature is currently in a "semi-broken" state and may not work reliably for all stream types.