Archives YouTube channels by automatically uploading their videos to archive.org
Project description
Internetarchive-YouTube
🚀 GitHub Action and CLI tool to archive YouTube channels by automatically uploading an entire YouTube channel to archive.org in few clicks.
📌 Global Requirements
- All you need is an Internet Archive account.
🔧 Usage
- ⚡️ To use this tool as a GitHub Action, jump to GitHub Action: Getting Started.
- 🧑💻 To use this tool as a command line interface (CLI), jump to CLI: Getting Started.
⚡️ GitHub Action: Getting Started
Using internetarchive-youtube as a GitHub Action instructions
-
Enable the workflows in your fork.
- Create a backend database (or JSON bin).
- Add your Archive.org credentials to the repository's Actions secrets:
ARCHIVE_USER_EMAIL
ARCHIVE_PASSWORD
- Add a list of the channels you want to archive as a
CHANNELS
secret to the repository's Actions secrets:
The CHANNELS
secret should be formatted like this example:
CHANNEL_NAME: CHANNEL_URL
FOO: FOO_CHANNEL_URL
FOOBAR: FOOBAR_CHANNEL_URL
SOME_CHANNEL: SOME_CHANNEL_URL
Don't add any quotes around the name or the URL, and make sure to keep one space between the colon and the URL.
- Add the database secret(s) to the repository's Actions secrets:
If you picked option 1 (MongoDB), add this secret:
MONGODB_CONNECTION_STRING
The value of the secret is the database conneciton string.
If you picked option 2 (JSON bin), add this additional secret:
JSONBIN_KEY
The value of this secret is the MASTER KEY token you copied from JSONbin.
- Run the workflow under
Actions
manually, or wait for it to run automatically every 6 hours.
That's it!
🧑💻 CLI: Getting Started
Using internetarchive-youtube as a CLI tool instructions
Requirements:
⬇️ Installation:
pip install internetarchive-youtube
# Install and login to internetarchive
pip install internetarchive
ia configure
🗃️ Backend database:
-
Create a backend database (or JSON bin) to track the download/upload overall progress.
-
If you choose MongoDB, export the connection string as an environment variable:
export MONGODB_CONNECTION_STRING=mongodb://username:password@host:port
# or add it to your shell configuration file:
echo "MONGODB_CONNECTION_STRING=$MONGODB_CONNECTION_STRING" >> "$HOME/.$(basename $SHELL)rc"
source "$HOME/.$(basename $SHELL)rc"
- If you choose JSONBin, export the master key as an environment variable:
export JSONBIN_KEY=xxxxxxxxxxxxxxxxx
# or add it to your shell configuration file:
echo "JSONBIN_KEY=$JSONBIN_KEY" >> "$HOME/.$(basename $SHELL)rc"
source "$HOME/.$(basename $SHELL)rc"
⌨️ Usage:
usage: ia-yt [-h] [-p PRIORITIZE] [-s SKIP_LIST] [-f] [-t TIMEOUT] [-n] [-a] [-c CHANNELS_FILE] [-S] [-C]
optional arguments:
-h, --help show this help message and exit
-p PRIORITIZE, --prioritize PRIORITIZE
Comma-separated list of channel names to prioritize
when processing videos
-s SKIP_LIST, --skip-list SKIP_LIST
Comma-separated list of channel names to skip
-f, --force-refresh Refresh the database after every video (Can slow down
the workflow significantly, but is useful when running
multiple concurrent jobs)
-t TIMEOUT, --timeout TIMEOUT
Kill the job after n hours (default: 5.5)
-n, --no-logs Don't print any log messages
-a, --add-channel Add a channel interactively to the list of channels to
archive
-c CHANNELS_FILE, --channels-file CHANNELS_FILE
Path to the channels list file to use if the
environment variable `CHANNELS` is not set (default:
~/.yt_channels.txt)
-S, --show-channels Show the list of channels in the channels file
-C, --create-collection
Creates/appends to the backend database from the
channels list
🏗️ Creating A Backend Database
Creating A Backend Database instructions
- Option 1: MongoDB (recommended).
- Self-hosted (see: Alyetama/quick-MongoDB or dockerhub image).
- Free cloud database on Atlas.
- Option 2: JSON bin (if you want a quick start).
- Sign up to JSONBin here.
- Click on
VIEW MASTER KEY
, then copy the key.
📝 Notes
- Information about the
MONGODB_CONNECTION_STRING
can be found here. - Jobs can run for a maximum of 6 hours, so if you're archiving a large channel, the job might die, but it will resume in a new job when it's scheduled to run.
- Instead of raw text, you can pass a file path or a file URL with a list of channels formatted as
CHANNEL_NAME: CHANNEL_URL
or in JSON format{"CHANNEL_NAME": "CHANNEL_URL"}
.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for internetarchive-youtube-0.2.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 75214de648cb01bad6714f84a935e82e563c4b2d8f7fbdb084cf4fefadae3bf5 |
|
MD5 | 254f9ccd17f8035e5589f79986003ab8 |
|
BLAKE2b-256 | 93d2bfdd932399e7125bc58e251e414c38915e765032c027aa89163ef8b1c6a6 |
Hashes for internetarchive_youtube-0.2.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9e8ad60f42721d8f5dd2052f680ac49610112e1f2349f2b1cbb576c0f518417a |
|
MD5 | 904a703af6ed3b1a2a26b37f02520954 |
|
BLAKE2b-256 | 40f3793616ccaacafd620ee8f23e3abe9938618df0233c3f7090d593390971a7 |