Skip to main content

Crochet-based blocking API for Scrapy.

Project description


Crochet-based blocking API for Scrapy.

This module provides function helpers to run Scrapy in a blocking fashion. See the scrapydo-overview.ipynb notebook for a quick overview of this module.


Using pip:

pip install scrapydo

Using conda:

conda install -c scrapydo


The function scrapydo.setup must be called once to initialize the reactor.


import scrapydo


# Enable logging display
import logging

# Fetch a single URL.
response = scrapydo.fetch("")

# Crawl an URL with given callback.
def parse_page(response):
    yield {
        'title': response.css('title').extract(),
        'url': response.url,
    for href in response.css('a::attr(href)'):
        url = response.urljoin(href)
        yield Request(url, callback=parse_page)

items = scrapydo.crawl('', callback)

# Run an existing spider class.
spider_args = {'foo': 'bar'}
items = scrapydo.run_spider(MySpider, **spider_args)

Available Functions

Initialize reactor.
scrapydo.fetch(url, spider_cls=DefaultSpider, capture_items=True, return_crawler=False, settings=None, timeout=DEFAULT_TIMEOUT)
Fetches an URL and returns the response.
scrapydo.crawl(url, callback, spider_cls=DefaultSpider, capture_items=True, return_crawler=False, settings=None, timeout=DEFAULT_TIMEOUT)
Crawls an URL with given callback and returns the scraped items.
scrapydo.run_spider(spider_cls, capture_items=True, return_crawler=False, settings=None, timeout=DEFAULT_TIMEOUT, **kwargs)
Runs a spider and returns the scraped items.
highlight(code, lexer='html', formatter='html', output_wrapper=None)
Highlights given code using pygments. This function is suitable for use in a IPython notebook.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for scrapydo, version 0.2.1
Filename, size File type Python version Upload date Hashes
Filename, size scrapydo-0.2.1.tar.gz (4.9 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page