Skip to main content

Patu is a small spider

Project description

Patu

A small spider, useful for checking a site for 404s and 500s. Patu requires httplib2 and lxml:

pip install -U httplib2 lxml

Quick Usage

To see available options:

patu.py –help

To spider an entire site using 5 workers, only showing errors:

patu.py –spiders=5 www.example.com

To spider, stopping after the first level of links:

patu.py –depth=1 www.example.com

To get a list of every linked page on a site:

patu.py –generate www.example.com > urls.txt

Instead of spidering for URLs, use a file instead and show all responses:

patu.py –input=urls.txt –verbose www.example.com

Format of URLs File

The output produced by <code>–generate</code> is formatted like so:

FIRST_URL<TAB>None LINK1<TAB>REFERER LINK2<TAB>REFERER

<code>–input</code> can take a file of that format, or one URL per line with no referer. <code>–input=-</code> reads from stdin.

Testing

Patu uses Nose for testing. To install Nose and test:

pip install -U nose nosetests

Project details


Release history Release notifications | RSS feed

This version

0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for patu, version 0.1
Filename, size File type Python version Upload date Hashes
Filename, size patu-0.1.tar.gz (9.1 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page