scrapy-boilerplate
scrapy-boilerplate is a small set of utilities for Scrapy to simplify writing low-complexity spiders that are very common in small and one-off projects.
It requires Scrapy (>= 0.16) and has been tested using python 2.7. Additionally, PyQuery is required to run the scripts in the examples directory.
Usage Guide
Items
Standard item definition:
from scrapy.item import Item, Field
class BaseItem(Item):
url = Field()
crawled = Field()
class UserItem(BaseItem):
name = Field()
about = Field()
location = Field()
class StoryItem(BaseItem):
title = Field()
body = Field()
user = Field()Becomes:
from scrapy_boilerplate import NewItem
BaseItem = NewItem('url crawled')
UserItem = NewItem('name about location', base_cls=BaseItem)
StoryItem = NewItem('title body user', base_cls=BaseItem)BaseSpider
Standard spider definition:
from scrapy.spider import BaseSpider
class MySpider(BaseSpider):
name = 'my_spider'
start_urls = ['http://example.com/latest']
def parse(self, response):
# do stuffBecomes:
from scrapy_boilerplate import NewSpider
MySpider = NewSpider('my_spider')
@MySpider.scrape('http://example.com/latest')
def parse(spider, response):
# do stuffCrawlSpider
Standard crawl-spider definition:
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
class MySpider(CrawlSpider):
name = 'my_spider'
start_urls = ['http://example.com']
rules = (
Rule(SgmlLinkExtractor('category\.php'), follow=True),
Rule(SgmlLinkExtractor('item\.php'), callback='parse_item'),
)
def parse_item(self, response):
# do stuffBecomes:
from scrapy_boilerplate import NewCrawlSpider
MySpider = NewCrawlSpider('my_spider')
MySpider.follow('category\.php')
@MySpider.rule('item\.php')
def parse_item(spider, response):
# do stuffRunning Helpers
Single-spider running script:
# file: my-spider.py
# imports omitted ...
class MySpider(BaseSpider):
# spider code ...
if __name__ == '__main__':
from scrapy_boilerplate import run_spider
custom_settings = {
# ...
}
spider = MySpider()
run_spider(spider, custom_settings)Multi-spider script with standard crawl command line options:
# file: my-crawler.py
# imports omitted ...
class MySpider(BaseSpider):
name = 'my_spider'
# spider code ...
class OtherSpider(CrawlSpider):
name = 'other_spider'
# spider code ...
if __name__ == '__main__':
from scrapy_boilerplate import run_crawler, SpiderManager
custom_settings = {
# ...
}
SpiderManager.register(MySpider)
SpiderManager.register(OtherSpider)
run_crawler(custom_settings)1 maintainer
Metadata
1 maintainer
Release files for scrapy-boilerplate 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| scrapy-boilerplate-0.2.1.tar.gz | 5.0 kB | Details |
Release files / scrapy-boilerplate-0.2.1.tar.gz
| Download URL | scrapy-boilerplate-0.2.1.tar.gz |
|---|---|
| Size | 5.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
49860bcb31ad8d7516e9f322b89b34ae708b4f373b588477cbb65eee3a273078
|
|
BLAKE2b-256 checksum How to use checksums |
654f8901d1fc946dc6bc27e8abe472d298db36da18db3e63793938383499dd6a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
