Clark University, Package for YouTube crawler and cleaning data
Project description
clarku-youtube-crawler
Clark University YouTube crawler and JSON decoder for YouTube json. Please read documentation in DOCS
Pypi page: "https://pypi.org/project/clarku-youtube-crawler/"
Installing
To install,
pip install clarku-youtube-crawler
The crawler needs multiple other packages to function.
If missing requirements (I already include all dependencies so it shouldn't happen), download requirements.txt
.
Navigate to the folder where it contains requirements.txt and run
pip install -r requirements.txt
YouTube API Key
- Go to https://cloud.google.com/, click console, and create a project. Under Credentials, copy the API key.
- In your project folder, create a "DEVELOPER_KEY.txt" file (must be this file name) and paste your API key.
- You can use multiple API keys by putting them on different lines in DEVELOPER_KEY.txt.
- The crawler will use up all quotas of one key and try next one, until all quotas are used up.
Example usage
Case 1: crawl videos by keywords,
# your_script.py
import clarku_youtube_crawler as cu
test = cu.RawCrawler()
test.build()
test.crawl("any_keyword",start_date=14, start_month=12, start_year=2020, day_count=2)
test.merge_to_workfile()
test.crawl_videos_in_list(comment_page_count=1)
test.merge_all(save_to='FINAL_merged_all.json')
Case 2: crawl a videos by a list of ids specified by videoId column in an input CSV
import clarku_youtube_crawler as cu
test = cu.RawCrawler()
test.build()
test.crawl_videos_in_list(video_list_workfile="video_list.csv", comment_page_count=1, search_key="LGBTQ")
test.merge_all(save_to='all.json')
Case 3: Get a list of channels crawled by keywords. I.e. get all videos in channels who have a video in a specific topic. You need to run case 1 first to generate a raw video JSON file (FINAL_merged_all.json). This code will find all unique channels in the video JSON and generate a list of channels. Then it will get all videos of all channels.
import clarku_youtube_crawler as cu
channel = cu.ChannelCrawler()
channel.build()
channel.setup_channel(filename='FINAL_merged_all.json', subscriber_cutoff=0, keyword="")
channel.crawl()
channel.crawl_videos_in_list(comment_page_count=3) #100 comments per page, 3 page will crawl first 300 comments
channel.merge_all(save_to='file_of_merged_json.json')
Case 4: You already have a list of channels. You want to crawl all videos of the channels in the list:
import clarku_youtube_crawler as cu
channel = cu.ChannelCrawler()
channel.build()
channel.crawl(filename='channel_list.csv', channel_header="channelId")
channel.crawl_videos_in_list(comment_page_count=3) #100 comments per page, 3 page will crawl first 300 comments
channel.merge_all(save_to='file_of_merged_json.json')
To convert the video JSONs to csv:
jsonn = cu.JSONDecoder()
jsonn.load_json("FINAL_merged_all.json")
Changelog
Version 0.0.1->0.0.3
This is beta without testing since python packaging is a pain. Please don't install these versions.
Version 0.0.5
Finally figured out testing. It works okay. More documentation to come.
Version 0.0.6
Stable release only for RawCrawler
feature
Version 1.0.0
Version 1.0.1
I think this might be our first full stable release.
Version 1.0.1.dev
Pre-release
Added different file types for ChannelCrawler. Added documentation
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for clarku_youtube_crawler-1.1.15.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9d3bbf2fb37bb681d1096a59beb65e9ea9466a966b36ff37235bb6798a7a0b3e |
|
MD5 | 7d7555970dbb40334a2e4d27b2afac9d |
|
BLAKE2b-256 | e3f60a3cf30fdb804b71812b595155f2b7c98e0531f2808dc72f4d6eece9ffb4 |
Hashes for clarku_youtube_crawler-1.1.15-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 984bf4fd6c106733e418ef7e07373cb2420f9a6f7da2a587184df7cd2dc6a996 |
|
MD5 | bf96eb2be843922a40bbad5941ba78ae |
|
BLAKE2b-256 | 2025fe5781b87eab70811a4039ae5ed952f5b1936f707844e3f978ca5218846e |