Skip to main content

Web spider and fetcher

Project description

https://badge.fury.io/py/spiderfetch.png https://travis-ci.org/numerodix/spiderfetch.png?branch=master

Installation

$ pip install spiderfetch
$ spiderfetch

Usage

Fetching

Fetch all urls matching 2008.*.ogg from a page:

$ spiderfetch http://www.fosdem.org/2008/media/video 2008.*ogg

To dump the urls to a file instead of fetching:

$ spiderfetch http://www.fosdem.org/2008/media/video 2008.*ogg --dump > urls

Spidering

Spider a site to depth 3 while pausing 2 seconds between fetches. The urls that will be considered when spidering must match .*:

$ spiderfetch --host http://en.wikipedia.org --depth 3 --pause 2 '.*'

Project details


Release history Release notifications

This version
History Node

0.4.9

History Node

0.4.8

History Node

0.4.7

History Node

0.4.6

History Node

0.4.5

History Node

0.4.4

History Node

0.4.3

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
spiderfetch-0.4.9-py2.py3-none-any.whl (24.0 kB) Copy SHA256 hash SHA256 Wheel 2.7 Apr 18, 2014
spiderfetch-0.4.9.tar.gz (18.0 kB) Copy SHA256 hash SHA256 Source None Apr 18, 2014

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page