YoutubeVideoCrawler 0.1.4

A Python module that crawls YouTube videos and comments based on keywords.

🎥 YoutubeVideoCrawler

A Python module for scalable YouTube data collection using the YouTube Data API v3.
It supports keyword-based crawling, comment extraction, intelligent rate limiting, and structured storage (JSON + CSV).

---

🚀 Features

• 🔍 Crawl YouTube videos by keywords
• 🧠 Supports multi-keyword and incremental crawling
• 💬 Optionally fetch video comments
• 🧾 Stores all video metadata in JSON + indexed CSV
• 🧰 Automatically skips duplicate videos
• 💤 Rate-limit friendly (safe_sleep)
• 📜 Flexible logger: print to console, file, or both

---

📦 Installation

pip install YoutubeVideoCrawler

(If installing locally from source after modification:)

pip install .

---

🔑 Requirements

Before running, you must have a YouTube Data API key:

1. Visit Google Cloud Console
2. Create a project → Enable the YouTube Data API v3
3. Generate an API key

---

🧩 Basic Usage

from YoutubeVideoCrawler.crawler import YouTubeCrawler

# Initialize the crawler
crawler = YouTubeCrawler(api_key="YOUR_YOUTUBE_API_KEY")

# Start crawling
crawler.run(
    keywords=["software engineering", "machine learning", "distributed systems"],
    fetch_comments=False,
    max_per_keyword=100
)

This will:

• Crawl up to 100 videos for each keyword
• Store video metadata in videos.json
• Maintain an index in video_index.csv

---

📂 Storage Output

After running, the module creates the following files:

File	Description
videos.json	Full JSON dump of all fetched video metadata
video_index.csv	Lightweight index (videoId, title) for quick reference
token.csv	Tracks how many videos have been fetched per keyword
app.log	Logger file recording crawl progress

---

⚙️ Advanced Usage

🧠 Load keywords from file

Create a file keywords.txt:

software engineering
deep learning
data visualization

Then run:

crawler.run(keyword_file="keywords.txt", max_per_keyword=50)

---

🪣 Multi-keyword mode

Search all keywords in a single query (logical OR):

crawler.run(
    keywords=["AI", "ML", "DL"],
    search_mode="multi",
    max_per_keyword=200
)

---

💬 Fetch comments

Enable fetching comments for each video:

crawler.run(
    keywords=["open source software"],
    fetch_comments=True,
    max_per_keyword=20
)

Comments are stored within the corresponding video entry in videos.json.

---

🕒 Resume partially completed crawl

The crawler automatically:

• Detects existing videos in video_index.csv
• Skips duplicates
• Continues from where it left off

---

🧱 Module Structure

YoutubeVideoCrawler/
│
├── crawler.py      # Core crawling logic using YouTube API
├── storage.py      # Persistent storage manager (JSON + CSV)
└── utils.py        # Logging, sleep, and helper functions

---

🧰 Utility Functions

get_logger(name="YouTubeCrawler", log_file="app.log")

Creates a flexible logger that can print to console, file, or both:

logger.info("This message prints to console and file.")
logger.info("This prints only to file.", print_to="file")

safe_sleep(seconds, logger)

Sleep while logging to respect API rate limits.

keyword_loader(file_path)

Loads keyword list from file, skipping lines starting with #.

---

📊 Example Output

Example video_index.csv:

videoId,title
XyZ12345,Introduction to Software Engineering
AbC67890,Machine Learning Basics

Example JSON entry:

{
  "id": "XyZ12345",
  "snippet": {
    "title": "Introduction to Software Engineering",
    "publishedAt": "2025-01-01T12:00:00Z"
  },
  "statistics": {
    "viewCount": "15342",
    "likeCount": "876"
  },
  "keyword": "software engineering",
  "comments": []
}

---

🧩 Command-Line Example (optional script)

You can create a small script crawl_youtube.py:

from yt_video_crawler import YouTubeCrawler

if __name__ == "__main__":
    crawler = YouTubeCrawler(api_key="YOUR_API_KEY")
    crawler.run(keyword_file="keywords.txt", fetch_comments=False, max_per_keyword=50)

Run it:

python crawl_youtube.py

---

🧑‍💻 Developer Notes

• Respects YouTube API quota and sleep intervals.
• Modular design for easy integration with data pipelines.
• Extend Storage class for custom database backends (e.g., MongoDB, SQLite).
• Log verbosity can be controlled using print_to argument.

---

📜 License

MIT License © 2025 Md. Masud Mazumder

---

🧠 Acknowledgments

• Google API Python Client
• tqdm for progress bars
• pandas for data handling

---

🤝 Contributing

Pull requests are welcome!