scrapy-boilerplate
scrapy-boilerplate is a small set of utilities for Scrapy to simplify writing low-complexity spiders that are very common in small and one-off projects.
It requires Scrapy (>= 0.16) and has been tested using python 2.7. Additionally, PyQuery is required to run the scripts in the examples directory.
Usage Guide
Items
Standard item definition:
from scrapy.item import Item, Field
class BaseItem(Item):
url = Field()
crawled = Field()
class UserItem(BaseItem):
name = Field()
about = Field()
location = Field()
class StoryItem(BaseItem):
title = Field()
body = Field()
user = Field()Becomes:
from scrapy_boilerplate import NewItem
BaseItem = NewItem('url crawled')
UserItem = NewItem('name about location', base_cls=BaseItem)
StoryItem = NewItem('title body user', base_cls=BaseItem)BaseSpider
Standard spider definition:
from scrapy.spider import BaseSpider
class MySpider(BaseSpider):
name = 'my_spider'
start_urls = ['http://example.com/latest']
def parse(self, response):
# do stuffBecomes:
from scrapy_boilerplate import NewSpider
MySpider = NewSpider('my_spider')
@MySpider.scrape('http://example.com/latest')
def parse(spider, response):
# do stuffCrawlSpider
Standard crawl-spider definition:
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
class MySpider(CrawlSpider):
name = 'my_spider'
start_urls = ['http://example.com']
rules = (
Rule(SgmlLinkExtractor('category\.php'), follow=True),
Rule(SgmlLinkExtractor('item\.php'), callback='parse_item'),
)
def parse_item(self, response):
# do stuffBecomes:
from scrapy_boilerplate import NewCrawlSpider
MySpider = NewCrawlSpider('my_spider')
MySpider.follow('category\.php')
@MySpider.rule('item\.php')
def parse_item(spider, response):
# do stuffRunning Helpers
Single-spider running script:
# file: my-spider.py
# imports omitted ...
class MySpider(BaseSpider):
# spider code ...
if __name__ == '__main__':
from scrapy_boilerplate import run_spider
custom_settings = {
# ...
}
spider = MySpider()
run_spider(spider, custom_settings)Multi-spider script with standard crawl command line options:
# file: my-crawler.py
# imports omitted ...
class MySpider(BaseSpider):
name = 'my_spider'
# spider code ...
class OtherSpider(CrawlSpider):
name = 'other_spider'
# spider code ...
if __name__ == '__main__':
from scrapy_boilerplate import run_crawler, SpiderManager
custom_settings = {
# ...
}
SpiderManager.register(MySpider)
SpiderManager.register(OtherSpider)
run_crawler(custom_settings)1 maintainer
Metadata
1 maintainer
Release files for scrapy-boilerplate 0.2.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| scrapy-boilerplate-0.2.1.tar.gz | 5.0 kB | Without provenance | Details for scrapy-boilerplate-0.2.1.tar.gz |
Release files / scrapy-boilerplate-0.2.1.tar.gz
| Download URL | scrapy-boilerplate-0.2.1.tar.gz |
|---|---|
| Size | 5.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
49860bcb31ad8d7516e9f322b89b34ae708b4f373b588477cbb65eee3a273078
|
|
BLAKE2b-256 checksum How to use checksums |
654f8901d1fc946dc6bc27e8abe472d298db36da18db3e63793938383499dd6a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for scrapy-boilerplate 0.2.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
