Skip to main content

A Powerful Spider System in Python

Project description

pyspider [![Build Status](]( [![Coverage Status](](

A Powerful Spider(Web Crawler) System in Python. [Try It Now!](

- Write script in python with powerful API
- Powerful WebUI with script editor, task monitor, project manager and result viewer
- MySQL, MongoDB, SQLite as database backend
- Javascript pages supported!
- Task priority, retry, periodical and recrawl by age or marks in index page (like update time)
- Distributed architecture

Sample Code:

from libs.base_handler import *

class Handler(BaseHandler):
this is a sample handler
@every(minutes=24*60, seconds=0)
def on_start(self):
self.crawl('', callback=self.index_page)

def index_page(self, response):
for each in response.doc('a[href^="http://"]').items():
self.crawl(each.attr.href, callback=self.detail_page)

def detail_page(self, response):
return {
"url": response.url,
"title": response.doc('title').text(),



* python2.6/7 (windows is not supported currently)
* `pip install --allow-all-external -r requirements.txt`
* `./` , visit [http://localhost:5000/](http://localhost:5000/)

if ubuntu: `apt-get install python python-dev python-distribute python-pip libcurl4-openssl-dev libxml2-dev libxslt1-dev python-lxml`

[Running with Docker](


* [Quickstart](
* [API Reference](
* more documents: [Wiki](


* Use It, Open [Issue](, PR is welcome.
* [Discuss](, [Document](

Licensed under the Apache License, Version 2.0

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pyspider, version 0.3.0.dev5
Filename, size File type Python version Upload date Hashes
Filename, size pyspider-0.3.0.dev5.tar.gz (65.7 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page