Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Crochet-based blocking API for Scrapy.

Project description

ScrapyDo

Crochet-based blocking API for Scrapy.

This module provides function helpers to run Scrapy in a blocking fashion. See the scrapydo-overview.ipynb notebook for a quick overview of this module.

Installation

Using pip:

pip install scrapydo

Usage

The function scrapydo.setup must be called once to initialize the reactor.

Example:

import scrapydo
scrapydo.setup()

scrapydo.default_settings.update({
    'LOG_LEVEL': 'DEBUG',
    'CLOSESPIDER_PAGECOUNT': 10,
})

# Enable logging display
import logging
logging.basicConfig(level=logging.DEBUG)

# Fetch a single URL.
response = scrapydo.fetch("http://example.com")

# Crawl an URL with given callback.
def parse_page(response):
    yield {
        'title': response.css('title').extract(),
        'url': response.url,
    }
    for href in response.css('a::attr(href)'):
        url = response.urljoin(href)
        yield Request(url, callback=parse_page)

items = scrapydo.crawl('http://example.com', callback)

# Run an existing spider class.
spider_args = {'foo': 'bar'}
items = scrapydo.run_spider(MySpider, **spider_args)

Available Functions

scrapydo.setup()
Initialize reactor.
scrapydo.fetch(url, spider_cls=DefaultSpider, capture_items=True, return_crawler=False, settings=None, timeout=DEFAULT_TIMEOUT)
Fetches an URL and returns the response.
scrapydo.crawl(url, callback, spider_cls=DefaultSpider, capture_items=True, return_crawler=False, settings=None, timeout=DEFAULT_TIMEOUT)
Crawls an URL with given callback and returns the scraped items.
scrapydo.run_spider(spider_cls, capture_items=True, return_crawler=False, settings=None, timeout=DEFAULT_TIMEOUT, **kwargs)
Runs a spider and returns the scraped items.
highlight(code, lexer='html', formatter='html', output_wrapper=None)
Highlights given code using pygments. This function is suitable for use in a IPython notebook.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for scrapydo, version 0.2.2
Filename, size File type Python version Upload date Hashes
Filename, size scrapydo-0.2.2.tar.gz (4.9 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page