Skip to main content

A Powerful Spider System in Python

Project description

pyspider [![Build Status][Build Status]][Travis CI] [![Coverage Status][Coverage Status]][Coverage] [![Try It Now!][Try]][Demo]

A Powerful Spider(Web Crawler) System in Python. **[TRY IT NOW!][Demo]**

- Write script in python with powerful API
- Python 2&3
- Powerful WebUI with script editor, task monitor, project manager and result viewer
- Javascript pages supported!
- MySQL, MongoDB, SQLite as database backend
- Task priority, retry, periodical, recrawl by age and more
- Distributed architecture

Documentation: [](

Sample Code

from pyspider.libs.base_handler import *

class Handler(BaseHandler):
crawl_config = {

@every(minutes=24 * 60)
def on_start(self):
self.crawl('', callback=self.index_page)

@config(age=10 * 24 * 60 * 60)
def index_page(self, response):
for each in response.doc('a[href^="http"]').items():
self.crawl(each.attr.href, callback=self.detail_page)

def detail_page(self, response):
return {
"url": response.url,
"title": response.doc('title').text(),

[![Demo][Demo Img]][Demo]


* `pip install pyspider`
* run command `pyspider`, visit [http://localhost:5000/](http://localhost:5000/)



* Use It
* Open [Issue], send PR
* [User Group]


### v0.3.0 (current)

- [x] as a package
- [x] parameters
- [x] sortable projects list #12
- [x] Postgresql Supported via SQLAlchemy (with the power of SQLAlchemy, pyspider also support Oracle, SQL Server, etc)
- [x] benchmarking
- [x] python3 support
- [x] documents
- [x] tutorial
- [ ] pypi release version

### v0.4.0

- [ ] local mode, load script from file.
- [ ] works as a framework (all components running in one process, no threads)
- [ ] shell mode like `scrapy shell`
- [ ] a visual scraping interface like [portia](

### more

- [ ] edit script with vim via [WebDAV](
- [ ] in-browser debugger like [Werkzeug](

Licensed under the Apache License, Version 2.0

[Build Status]:
[Travis CI]:
[Coverage Status]:
[Demo Img]:
[User Group]:

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pyspider, version 0.3.0b1
Filename, size File type Python version Upload date Hashes
Filename, size pyspider-0.3.0b1.tar.gz (74.9 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page