macrawlon 0.1

(a Multi-processing Audiovisual CRAWLer collectiON)

About

A package for crawling and downloading YouTube videos. As multiple datasets that are introduced only provide the ids of videos without a download script, obtaining the video files may be difficult. This project aims to provide a general solution is such cases by downloading either the video or audio from ids specified by a dataset. It also aims to speed up processing though enabling multiple threads to run in parallel. The video resolution is user set in order to speed-up downloading and to limit the on-disk dataset size.

Currently only video-only or audio-only files are downloaded (the next update/version will allow to also download videos with audio).

Package Requirements

This is the list of the required packages:

• pandas
• pafy
• ffmpeg
• youtube-dl
• tqdm

They can all be downloaded with:

$ pip install pandas pafy tqdm

CSV Dataset file

The package assumes that the following headers are included in the .csv file that includes the YouTube ids:

The name of the headers do not need to match exactly but the data needs to include the id, start time end time or duration.

Usage

The main function used to download files is called download() as is located at the youtube_audio_and_video_downloader.py. You can simply call it by first importing it:

from macrawlon import download
#or
from youtube_audio_and_video_downloader import download

download(
  csv_dir=my_csv_dir,
  download_dir=my_down_dir,
  modality='video',
  resolutions=my_res_list,
  id_idx = 0,
  start_idx = 1,
  end_idx = None,
  duration=10,
  workers=5
  )

The function takes the following arguments:

Installation through git

Please make sure, Git is installed in your machine:

$ sudo apt-get update
$ sudo apt-get install git
$ git clone https://github.com/macrawlon/macrawlon.git
$ cd macrawlon
$ pip install .

You can then use it as any other package installed through pip.

Installation through pip

The latest stable release is also available for download through pip

$ pip install macrawlon

Licence

MIT