DatasetScraper

Tool to create image datasets for machine learning problems by scraping search engines like Google, Bing and Baidu.

Features:

• Search engine support: Google, Bing, Baidu. (in-production): Yahoo, Yandex, Duckduckgo
• Image format support: jpg, png, svg, gif, jpeg
• Fast multiprocessing enabled scraper
• Very fast multithreaded downloader
• Data verification after download for assertion of image files

Installation

• COMING SOON on pypi

Usage:

• Import from datasetscraper import Scraper

• Defaults

obj = Scraper()
urls = obj.fetch_urls('kiniro mosaic')
obj.download(urls, directory='kiniro_mosaic/')

• Specify a search engine

obj = Scraper()
urls = obj.fetch_urls('kiniro mosaic', engine=['google'])
obj.download(urls, directory='kiniro_mosaic/')

• Specify a list of search engines

obj = Scraper()
urls = obj.fetch_urls('kiniro mosaic', engine=['google', 'bing'])
obj.download(urls, directory='kiniro_mosaic/')

• Specify max images (default was 200)

obj = Scraper()
urls = obj.fetch_urls('kiniro mosaic', engine=['google', 'bing'], maxlist=[500, 300])
obj.download(urls, directory='kiniro_mosaic/')

FAQs

• Why aren't yandex, yahoo, duckduckgo and other search engines supported? They are hard to scrape, I am working on them and will update as soon as I can.

• I set maxlist=[500] why are only (x<500) images downloaded? There can be several reasons for this:

  • Search ran out: This happens very often, google/bing might not have enough images for your query
  • Slow internet: Increase the timeout (default is 60 seconds) as follows: obj.download(urls, directory='kiniro_mosaic/', timeout=100)

• How to debug? You can change the logging level while making the scraper object : obj = Scraper(logger.INFO)

TODO:

• More search engines
• Better debug
• Write documentation
• Text data? Audio data?