youtube-audio-dataset-collector 0.1.2

A comprehensive tool for downloading, processing, and transcribing audio from YouTube videos and playlists for machine learning datasets

YouTube Audio Dataset Collector

A comprehensive Python library for downloading, processing, and transcribing audio from YouTube videos and playlists for machine learning datasets.

Features

• YouTube Audio Download: Download audio from single videos or entire playlists using yt-dlp
• Audio Processing: Convert audio to 16kHz, 16-bit mono WAV format optimized for ML
• Intelligent Segmentation: Segment audio into 12-28 second chunks based on silence detection
• Multi-API Transcription: Transcribe audio using Gemini API with multiple key rotation for rate limit avoidance
• Parallel Processing: Multi-threaded processing for efficient handling of large datasets
• CSV Export: Export results in CSV format with audio_filepath and transcript columns

Installation

From PyPI (recommended)

pip install youtube-audio-dataset-collector

From Source

git clone https://github.com/yourusername/youtube-audio-dataset-collector.git
cd youtube-audio-dataset-collector
pip install -e .

System Dependencies

The library requires ffmpeg for audio processing:

Ubuntu/Debian:

sudo apt install ffmpeg

macOS:

brew install ffmpeg

Windows: Download from https://ffmpeg.org/download.html

Quick Start

Command Line Interface

# Basic usage
youtube-audio-collector --url "https://www.youtube.com/watch?v=VIDEO_ID" --output ./my_dataset

# With multiple API keys for better rate limits
youtube-audio-collector --url "https://www.youtube.com/playlist?list=PLAYLIST_ID" \
                       --output ./my_dataset \
                       --keys-file gemini_keys.txt \
                       --rotate-keys

# Advanced options
youtube-audio-collector --url "https://www.youtube.com/watch?v=VIDEO_ID" \
                       --output ./my_dataset \
                       --threads 8 \
                       --batch-size 6 \
                       --language en

Python API

from youtube_audio_collector import YouTubeAudioCollector

# Initialize with single API key
collector = YouTubeAudioCollector(
    output_dir="./my_dataset",
    api_key="your_gemini_api_key",
    threads=4
)

# Process a YouTube URL
output_csv = collector.process_url("https://www.youtube.com/watch?v=VIDEO_ID")
print(f"Dataset saved to: {output_csv}")

# Initialize with multiple API keys for better rate limits
collector = YouTubeAudioCollector(
    output_dir="./my_dataset",
    api_keys=["key1", "key2", "key3"],
    threads=8,
    batch_size=6
)

# Process a playlist
output_csv = collector.process_url(
    "https://www.youtube.com/playlist?list=PLAYLIST_ID",
    language="kn"  # Kannada
)

Individual Components

from youtube_audio_collector import (
    YouTubeDownloader, 
    AudioProcessor, 
    TranscriptionService,
    APIKeyManager
)

# Download audio
downloader = YouTubeDownloader(cookies_path="cookies.txt")
audio_files = downloader.download_audio(
    "https://www.youtube.com/watch?v=VIDEO_ID",
    "./downloads"
)

# Process audio
processor = AudioProcessor(sample_rate=16000)
segments = processor.segment_audio(
    audio_files[0],
    "./segments",
    min_length=12000,  # 12 seconds
    max_length=28000   # 28 seconds
)

# Transcribe with multiple API keys
api_manager = APIKeyManager(["key1", "key2", "key3"])
transcriber = TranscriptionService(api_key_manager=api_manager)

for segment in segments:
    transcript = transcriber.transcribe_audio(segment, language="en")
    print(f"{segment}: {transcript}")

Configuration

Environment Variables

Create a .env file in your project directory:

# Single API key
GEMINI_API_KEY=your_gemini_api_key

# Multiple API keys (for rate limit avoidance)
GEMINI_API_KEY_1=your_first_api_key
GEMINI_API_KEY_2=your_second_api_key
GEMINI_API_KEY_3=your_third_api_key

API Keys File

Create a text file with one API key per line:

AIzaSyD...key1
AIzaSyE...key2  
AIzaSyF...key3

Cookies for Restricted Content

To access age-restricted or private content:

1. Install a browser extension like Cookie Editor
2. Export YouTube cookies in Netscape format
3. Save as cookies.txt
4. Pass the file path to the library

Output Structure

The library organizes output in a structured format:

output_directory/
├── raw/                    # Downloaded audio files
│   ├── video1.wav
│   └── video2.wav
├── converted/              # Format-converted audio
│   ├── converted_video1.wav
│   └── converted_video2.wav  
├── segments/               # Segmented audio chunks
│   ├── video1_segment_001.wav
│   ├── video1_segment_002.wav
│   └── ...
└── transcriptions.csv      # Final dataset

The CSV contains two columns:

• audio_filepath: Path to audio segment
• transcript: Transcribed text

API Reference

YouTubeAudioCollector

Main class for the complete pipeline.

collector = YouTubeAudioCollector(
    output_dir="./output",           # Output directory
    prefix="dataset",                # File prefix
    cookies_path="cookies.txt",      # Cookies file path
    api_keys=["key1", "key2"],       # Multiple API keys
    api_key="single_key",            # Single API key
    sample_rate=16000,               # Audio sample rate
    threads=4,                       # Processing threads
    batch_size=4,                    # Transcription batch size
    calls_per_key_per_minute=15,     # API rate limit
    key_cooldown=60                  # Key cooldown period
)

APIKeyManager

Manages multiple API keys with intelligent rotation.

manager = APIKeyManager(
    api_keys=["key1", "key2", "key3"],
    calls_per_key_per_minute=15,
    cooldown_period=60
)

# Get next available key
key = manager.get_next_available_key()

# Mark key as unhealthy (e.g., after rate limit)
manager.mark_key_unhealthy(key)

YouTubeDownloader

Downloads audio from YouTube URLs.

downloader = YouTubeDownloader(cookies_path="cookies.txt")
files = downloader.download_audio(url, output_dir, prefix="dataset")

AudioProcessor

Processes and segments audio files.

processor = AudioProcessor(sample_rate=16000)

# Convert format
converted = processor.convert_format(audio_path, output_dir)

# Segment audio
segments = processor.segment_audio(
    audio_path, 
    output_dir,
    min_length=12000,    # 12 seconds
    max_length=28000,    # 28 seconds
    min_silence_len=500, # 500ms
    silence_thresh=-35   # -35dB
)

TranscriptionService

Transcribes audio using Gemini API.

# With API key manager
transcriber = TranscriptionService(api_key_manager=manager)

# With single key
transcriber = TranscriptionService(api_key="your_key")

# Transcribe
transcript = transcriber.transcribe_audio(audio_path, language="en")

Supported Languages

The library supports transcription in multiple languages:

• en - English
• kn - Kannada
• hi - Hindi
• es - Spanish
• fr - French
• And many more supported by Gemini API

Performance Tips

1. Use Multiple API Keys: Significantly improves throughput by avoiding rate limits
2. Optimize Thread Count: Use 4-8 threads for most systems
3. Batch Processing: Use batch sizes of 4-6 for optimal API usage
4. Storage: Use SSD storage for better I/O performance with large datasets

Legal Considerations

• Ensure you comply with YouTube's Terms of Service
• Only download content you have rights to use
• Respect content creators' intellectual property
• Consider fair use guidelines for research and educational purposes

Contributing

Contributions are welcome! Please read our contributing guidelines and submit pull requests to our GitHub repository.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

• Documentation: https://youtube-audio-dataset-collector.readthedocs.io/
• Issues: GitHub Issues
• Discussions: GitHub Discussions

Changelog

v1.0.0

• Initial release
• YouTube audio downloading with yt-dlp
• Intelligent audio segmentation
• Multi-API key support for Gemini transcription
• Parallel processing
• Command-line interface
• Python API