yt-fts - Youtube Full Text Search

yt-fts is a simple python script that uses yt-dlp to scrape all of a youtube channels subtitles and load them into an sqlite database that is searchable from the command line. It allows you to query a channel for specific key word or phrase and will generate time stamped youtube urls to the video containing the keyword.

• Blog Post
• Semantic Search (Experimental)

https://github.com/NotJoeMartinez/yt-fts/assets/39905973/6ffd8962-d060-490f-9e73-9ab179402f14

Installation

pip

pip install yt-fts

from source

git clone https://github.com/NotJoeMartinez/yt-fts
python3 -m venv .env
source .env/bin/activate
pip install -r requirements.txt
python3 -m yt-fts

Dependencies

This project requires yt-dlp installed globally. Platform specific installation instructions are available on the yt-dlp wiki.

pip

python3 -m pip install -U yt-dlp

MacOS/Homebrew

brew install yt-dlp

Windows/winget

winget install yt-dlp

Usage

Usage: yt-fts [OPTIONS] COMMAND [ARGS]...

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  delete              Delete a channel and all its data.
  download            Download subtitles from a specified YouTube channel.
  export              Export search results from a specified YouTube...
  generate-embeddings  Generate embeddings for a channel using OpenAI's...
  list                Lists channels saved in the database.
  search              Search for a specified text within a channel, a...
  semantic-search     Semantic search for specified text.
  show                Show transcripts for a video or list of videos...
  update              Updates a specified YouTube channel.

download

Download subtitles

Usage: yt-fts download [OPTIONS] CHANNEL_URL

  Download subtitles from a specified YouTube channel.

  You must provide the URL of the channel as an argument. The script will
  automatically extract the channel id from the URL.

Options:
  --channel-id TEXT         Optional channel id to override the one from the
                            url
  --language TEXT           Language of the subtitles to download
  --number-of-jobs INTEGER  Optional number of jobs to parallelize the run

Examples:

Basic download by url

yt-fts download "https://www.youtube.com/@TimDillonShow/videos"

Multithreaded download

yt-fts download --number-of-jobs 6 "https://www.youtube.com/@TimDillonShow/videos"

specify channel id

If download fails you can manually input the channel id with the --channel-id flag. The channel url should still be an argument

yt-fts download --channel-id "UC4woSp8ITBoYDmjkukhEhxg" "https://www.youtube.com/@TimDillonShow/videos" 

specify language

Languages are represented using ISO 639-1 language codes

yt-fts download --language de "https://www.youtube.com/@TimDillonShow/videos" 

list

List downloaded channels

Usage: yt-fts list [OPTIONS]

  Lists channels saved in the database.

  The (ss) next to channel name indicates that semantic search is enabled for
  the channel.

Options:
  --channel TEXT  Optional name or id of the channel to list

yt-fts list

output:

  id    count  channel_name         channel_url
----  -------  -------------------  ----------------------------------------------------
   1      265  The Tim Dillon Show  https://youtube.com/channel/UC4woSp8ITBoYDmjkukhEhxg
   2      688  Lex Fridman (ss)     https://youtube.com/channel/UCSHZKyawb77ixDdsGog4iWA
   3      434  Traversy Media       https://youtube.com/channel/UC29ju8bIPH5as8OGnQzwJyA

search

Search saved subtitles

Usage: yt-fts search [OPTIONS] SEARCH_TEXT

  Search for a specified text within a channel, a specific video, or across
  all channels.

Options:
  --channel TEXT  The name or id of the channel to search in. This is required
                  unless the --all or --video options are used.
  --video TEXT    The id of the video to search in. This is used instead of
                  the channel option.
  --all           Search in all channels.

• The search string does not have to be a word for word and match
• Use Id if you have channels with the same name or channels that have special characters in their name
• Search strings are limited to 40 characters.

Examples:

Search by channel

yt-fts search "life in the big city" --channel "The Tim Dillon Show"
# or 
yt-fts search "life in the big city" --channel 1  # assuming 1 is id of channel

output:

"Dennis would go hey life in the big city"

    Channel: The Tim Dillon Show
    Title: 154 - The 3 AM Episode - YouTube
    Time Stamp: 00:58:53.789
    Video ID: MhaG3Yfv1cU
    Link: https://youtu.be/MhaG3Yfv1cU?t=3530

Search all channels

yt-fts search "text to search" --all

Search in video

yt-fts search "text to search" --video [VIDEO_ID]

Advanced Search Syntax

The search string supports sqlite Enhanced Query Syntax. which includes things like prefix queries which you can use to match parts of a word.

yt-fts search "rea* kni* Mali*" --channel "The Tim Dillon Show" 

output:

"real knife fight down here in Malibu I"

    Channel: The Tim Dillon Show
    Title: #200 - Knife Fights In Malibu | The Tim Dillon Show - YouTube
    Time Stamp: 00:45:39.420
    Video ID: e79H5nxS65Q
    Link: https://youtu.be/e79H5nxS65Q?t=2736

export

Export search results to csv. Exported csv will have Channel Name,Video Title,Quote,Time Stamp,Link as it's headers

Usage: yt-fts export [OPTIONS] SEARCH_TEXT [CHANNEL]

  Export search results from a specified YouTube channel or from all channels
  to a CSV file.

  The results of the search will be exported to a CSV file. The file will be
  named with the format "{channel_id or 'all'}_{TIME_STAMP}.csv"

Options:
  --all   Export from all channels

Examples:

yt-fts export "life in the big city" "The Tim Dillon Show"

You can export from all channels in your database as well

yt-fts export "life in the big city" --all

update

Will update a channel with new subtitles if any are found.

Usage: yt-fts update [OPTIONS]

  Updates a specified YouTube channel.

  You must provide the ID of the channel as an argument. Keep in mind some
  might not have subtitles enabled. This command will still attempt to
  download subtitles as subtitles are sometimes added later.

Options:
  --channel TEXT            The name or id of the channel to update.
                            [required]
  --language TEXT           Language of the subtitles to download
  --number-of-jobs INTEGER  Optional number of jobs to parallelize the run

delete

Will delete a channel from your database

Usage: yt-fts delete [OPTIONS]

  Delete a channel and all its data.

  You must provide the name or the id of the channel you want to delete as an
  argument.

  The command will ask for confirmation before performing the deletion.

Options:
  --channel TEXT  The name or id of the channel to delete [required]

Examples:

yt-fts delete "The Tim Dillon Show"
# or
yt-fts delete 1 

show

Usage: yt-fts show [OPTIONS]

  Show transcripts for a video or list of videos from a channel

Options:
  -v, --video TEXT    The video id to show transcripts for
  -c, --channel TEXT  The name or id of the channel to show video list

Semantic Search via OpenAI embeddings API

The following commands are a work in progress but should enable semantic search. This requires that you have an openAI API key which you can learn more about that here.

Limitations

Keep in mind that generating embeddings will substantially grow the size of your subtitles database and will run slower due to the limitations of working with vectors in sqlite. When running semantic searches for the first time, API access is still required to generate embeddings for the search string. These search string embeddings are saved to a history table and won't require additional api requests after.

semantic-search

Usage: yt-fts semantic [OPTIONS] SEARCH_TEXT

  Semantic search for specified text.

  Before running this command, you must generate embeddings for the channel
  using the generate-embeddings command. This command uses OpenAI's embeddings 
  API to search for specified text. An OpenAI API key must be set as an
  environment variable OPENAI_API_KEY.

Options:
  --channel TEXT   channel name or id to search in
  --all            Search all semantic search enabled channels
  --limit INTEGER  top n results to return

generate-embedings

Usage: yt-fts generate-embedings [OPTIONS]

  Generate embeddings for a channel using OpenAI's embeddings API.

  Requires an OpenAI API key to be set as an environment variable
  OPENAI_API_KEY.

Options:
  --channel TEXT       The name or id of the channel to generate embeddings
                       for
  --open-api-key TEXT  OpenAI API key. If not provided, the script will
                       attempt to read it from the OPENAI_API_KEY environment
                       variable.