Skip to main content

A web bot to scrape images from websites.

Project description

A web bot to scrape images from websites.

Features

  • Supported platform: Linux (+Gnome) / Python 2.x.
  • Uses scrapy web crawling framework.
  • Maintains a database of all downloaded images to avoid duplicate downloads.
  • Optionally, it can scrape only under a particular url, e.g. scraping “http://website.com/albums/new” with this option will only download from new album.
  • You can specify minimum image size to be downloaded.
  • Scrapes through javascript popup links.
  • Live monitor window for displaying images as they are scraped.

Usage

  1. Scrape images from http://website.com:

    imagebot http://website.com
    
  2. Scrape images from http://website.com while allowing images from a cdn such as amazonaws.com (add multiple domains with comma separated list):

    imagebot http://website.com -d amazonaws.com
    
  3. Specify minimum size of image to be downloaded (width x height):

    imagebot http://website.com -s 300x300
    
  4. Stay under http://website.com/albums/new:

    imagebot http://website.com/albums/new -u http://website.com/albums/new
    
  5. Launch monitor windows for live images:

    imagebot http://website.com -m
    
  6. Set user-agent:

    imagebot http://website.com -a "my_imagebot(http://mysite.com)"
    
  7. For more options, get help:

    imagebot -h
    

Dependencies

  1. python-gi (Python GObject Introspection API)

    On Ubuntu:

    apt-get install python-gi
    
  2. scrapy (a powerful web crawling framework)

    It will be automatically installed by pip.

  3. Pillow (Python Imaging Library)

    It will be automatically installed by pip.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
imagebot-1.0.2.tar.gz (11.9 kB) Copy SHA256 hash SHA256 Source None Feb 1, 2015

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page