tinytinyscraper 0.1.1

A Python library that scrapes URLs and returns their contents. Special handling for YouTube URLs to extract transcripts and PDF files to extract text. Built with BeautifulSoup, Requests, YouTube-Transcript-API, and PyPDF.

URL Scraper

A Python library that scrapes URLs and returns their contents. Special handling for YouTube URLs (returns transcripts) and PDF files (extracts text).

Features

• 🎥 YouTube Transcript Extraction: Automatically detects YouTube URLs and extracts video transcripts
• 📄 PDF Text Extraction: Extracts text from PDF documents
• 🌐 Web Scraping: Scrapes text content from any webpage
• 🔄 Multiple Language Support: Get transcripts in different languages
• 📝 Detailed Transcript Data: Access timestamps and segments for YouTube videos
• 🔁 Automatic Retry: Failed requests are retried with exponential backoff
• 🚀 Simple API: Easy-to-use interface with sensible defaults

Installation

Install from PyPI:

pip install url-scraper

Or install from source for development:

# Clone the repository
git clone https://github.com/vunderkind/tinytinyscraper.git
cd url-scraper

# Install in development mode
pip install -e ".[dev]"

Quick Start

from tinytinyscraper import URLScraper

# Initialize the scraper
scraper = URLScraper()

# Scrape a YouTube video (returns detailed transcript data)
youtube_result = scraper.scrape("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
print(youtube_result['text'])  # Full transcript text
print(youtube_result['segments'])  # List of segments with timestamps

# Scrape a PDF document (returns extracted text)
pdf_text = scraper.scrape("https://example.com/document.pdf")
print(pdf_text)

# Scrape a regular webpage (returns text content)
webpage_text = scraper.scrape("https://example.com")
print(webpage_text)

# Get only text (works for YouTube, PDFs, and webpages)
text = scraper.scrape_text_only("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
print(text)

Usage Examples

PDF Document Extraction

from tinytinyscraper import URLScraper

scraper = URLScraper()

# Extract text from a PDF
text = scraper.scrape("https://www.irs.gov/pub/irs-pdf/fw4.pdf")
print(text)

# PDFs are automatically detected by .pdf extension or Content-Type header

YouTube Video Transcript

from tinytinyscraper import URLScraper

scraper = URLScraper()

# Get transcript with detailed information
result = scraper.scrape("https://www.youtube.com/watch?v=dQw4w9WgXcQ")

print(f"Full text: {result['text']}")
print(f"Language: {result['language']}")
print(f"Language code: {result['language_code']}")
print(f"Auto-generated: {result['is_generated']}")

# Access individual segments with timestamps
for segment in result['segments']:
    print(f"[{segment['start']:.2f}s] {segment['text']}")

Different YouTube URL Formats

The scraper supports various YouTube URL formats:

scraper = URLScraper()

# Standard watch URL
scraper.scrape("https://www.youtube.com/watch?v=VIDEO_ID")

# Short URL
scraper.scrape("https://youtu.be/VIDEO_ID")

# Embed URL
scraper.scrape("https://www.youtube.com/embed/VIDEO_ID")

Multi-Language Support

scraper = URLScraper()

# Try German first, then English
result = scraper.scrape(
    "https://www.youtube.com/watch?v=VIDEO_ID",
    languages=['de', 'en']
)

print(f"Retrieved transcript in: {result['language']}")

Regular Web Scraping

scraper = URLScraper()

# Scrape a webpage (automatically retries on failure)
text = scraper.scrape("https://www.example.com/article")
print(text)

# Custom timeout and retry settings
scraper = URLScraper(timeout=60, max_retries=5, retry_delay=2.0)
text = scraper.scrape("https://slow-website.com")

Retry Configuration

The scraper automatically retries failed requests with exponential backoff:

# Default: 3 retries with 1s, 2s, 4s delays
scraper = URLScraper()

# Custom retry settings for unreliable sites
scraper = URLScraper(
    max_retries=5,      # Try up to 5 times
    retry_delay=0.5     # Start with 0.5s, then 1s, 2s, 4s, 8s
)

# Disable retries (not recommended)
scraper = URLScraper(max_retries=1)

Retry behavior:

• Retries on: timeouts, connection errors, 429 (rate limit) errors
• No retry on: 4xx errors (except 429)
• Uses exponential backoff: delay × 2^attempt

Custom User Agent

scraper = URLScraper(
    user_agent="MyBot/1.0 (+http://mybot.com)"
)

text = scraper.scrape("https://example.com")

Text-Only Mode

scraper = URLScraper()

# Always returns a string, regardless of URL type
text = scraper.scrape_text_only("https://www.youtube.com/watch?v=VIDEO_ID")
print(text)  # Just the transcript text

text = scraper.scrape_text_only("https://example.com")
print(text)  # Just the webpage text

API Reference

URLScraper

Main scraper class.

Constructor

URLScraper(timeout=30, user_agent=None, max_retries=3, retry_delay=1.0)

Parameters:

• timeout (int): Request timeout in seconds (default: 30)
• user_agent (str, optional): Custom user agent string
• max_retries (int): Maximum number of retry attempts for failed requests (default: 3)
• retry_delay (float): Initial delay between retries in seconds, uses exponential backoff (default: 1.0)

Methods

scrape(url, languages=None)

Scrape content from a URL.

Parameters:

• url (str): The URL to scrape
• languages (list, optional): List of preferred language codes for YouTube transcripts (default: ['en'])

Returns:

• For YouTube URLs: Dictionary with keys:
  • text (str): Full transcript text
  • segments (list): List of transcript segments with text, start, and duration
  • language (str): Language name
  • language_code (str): Language code
  • is_generated (bool): Whether transcript is auto-generated
• For PDF files: String containing the extracted text
• For other URLs: String containing the text content

Raises:

• ValueError: If the URL is invalid
• Exception: If scraping fails

scrape_text_only(url, languages=None)

Scrape content and return only the text.

Parameters:

• url (str): The URL to scrape
• languages (list, optional): List of preferred language codes for YouTube transcripts

Returns:

• String containing the text content

Error Handling

from tinytinyscraper import URLScraper

scraper = URLScraper()

try:
    result = scraper.scrape("https://www.youtube.com/watch?v=INVALID")
except ValueError as e:
    print(f"Invalid URL: {e}")
except Exception as e:
    print(f"Error: {e}")

Common errors:

• TranscriptsDisabled: Video has transcripts disabled
• NoTranscriptFound: No transcript available in requested languages
• VideoUnavailable: Video is private or doesn't exist
• RequestException: Network or HTTP errors (automatically retried up to 3 times with exponential backoff)

Note: The scraper automatically retries failed requests up to 3 times with exponential backoff (1s, 2s, 4s) to handle temporary network issues.

Dependencies

• requests: HTTP library for making web requests
• beautifulsoup4: HTML/XML parser for web scraping
• lxml: Parser for BeautifulSoup
• pypdf: PDF text extraction
• youtube-transcript-api: YouTube transcript extraction

Development

Setup

# Clone the repository
git clone <your-repo-url>
cd yt-transcript

# Install in development mode with dev dependencies
pip install -e ".[dev]"

Running Tests

pytest

Code Formatting

black src/

Linting

flake8 src/

License

MIT License - feel free to use this library in your projects!

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Acknowledgments

• youtube-transcript-api for YouTube transcript extraction
• BeautifulSoup for HTML parsing
• Requests for HTTP requests