Skip to main content

Export scraped items of different types to multiple feeds.

Project description

This Scrapy extension exports scraped items of different types to multiple feeds. By default each item gets its own feed.

Installation

$ pip install scrapy-multifeedexporter

Configuration

You’ll have to switch the default FeedExporter with MultiFeedExporter by adding the following lines to the settings.py file of your spider:

from multifeedexporter import MultiFeedExporter

EXTENSIONS = {
    'scrapy.contrib.feedexport.FeedExporter': None,
    'multifeedexporter.MultiFeedExporter': 500,
}

# Automatically configure available item names from your module
MULTIFEEDEXPORTER_ITEMS = MultiFeedExporter.get_bot_items(BOT_NAME)

Usage

When calling scrapy crawl you need to use the %(item_name)s placeholder in the output file/URI name. The following calls to scrapy crawl demonstrate the placeholder:

$ scrapy crawl -o "spider_name_%(item_name)s.csv" -t csv spider_name
$ scrapy crawl -o "ftp://foo:bar@example.com/spider_name_%(item_name)s.csv" -t csv spider_name

If you omit the placeholder, all items will be placed in one file.

License

scrapy-multifeedexporter is published under MIT license

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
scrapy-multifeedexporter-0.1.1.tar.gz (2.7 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page