Skip to main content

A Powerful Spider System in Python

Project description

pyspider [![Build Status]][Travis CI] [![Coverage Status]][Coverage] [![Try]][Demo]

A Powerful Spider(Web Crawler) System in Python. **[TRY IT NOW!][Demo]**

- Write script in Python
- Powerful WebUI with script editor, task monitor, project manager and result viewer
- [MySQL](, [MongoDB](, [Redis](, [SQLite](, [Elasticsearch](; [PostgreSQL]( with [SQLAlchemy]( as database backend
- [RabbitMQ](, [Beanstalk](, [Redis]( and [Kombu]( as message queue
- Task priority, retry, periodical, recrawl by age, etc...
- Distributed architecture, Crawl Javascript pages, Python 2.{6,7}, 3.{3,4,5,6} support, etc...

Tutorial: [](
Documentation: [](
Release notes: [](

Sample Code

from pyspider.libs.base_handler import *

class Handler(BaseHandler):
crawl_config = {

@every(minutes=24 * 60)
def on_start(self):
self.crawl('', callback=self.index_page)

@config(age=10 * 24 * 60 * 60)
def index_page(self, response):
for each in response.doc('a[href^="http"]').items():
self.crawl(each.attr.href, callback=self.detail_page)

def detail_page(self, response):
return {
"url": response.url,
"title": response.doc('title').text(),

[![Demo][Demo Img]][Demo]


* `pip install pyspider`
* run command `pyspider`, visit [http://localhost:5000/](http://localhost:5000/)

**WARNING:** WebUI is open to the public by default, it can be used to execute any command which may harm your system. Please use it in an internal network or [enable `need-auth` for webui](

Quickstart: [](


* Use It
* Open [Issue], send PR
* [User Group]
* [中文问答](


### v0.4.0

- [ ] a visual scraping interface like [portia](

Licensed under the Apache License, Version 2.0

[Build Status]:
[Travis CI]:
[Coverage Status]:
[Demo Img]:
[User Group]:

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyspider-0.3.10.tar.gz (110.9 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page