scrapy-rss 0.1.2

RSS Tools for Scrapy Framework

Tools for easy RSS feed generating that contains each scraped item using Scrapy framework.

Package works with Python 2.7, 3.3, 3.4, 3.5 and 3.6.

Installation

• Install scrapy_rss using pip

  pip install scrapy_rss

  or using pip for specific interpreter, e.g.:

  pip3 install scrapy_rss

• or using directly setuptools:

  cd path/to/root/of/scrapy_rss
  python setup.py install

  or using setuptools for specific interpreter, e.g.:

  cd path/to/root/of/scrapy_rss
  python3 setup.py install

How To Use

Add parameters to the Scrapy project settings (settings.py file) or to the custom_settings attribute of the spider:

1. Add item pipeline that export items to rss feed:

   ITEM_PIPELINES = {
       # ...
       'scrapy_rss.pipelines.RssExportPipeline': 900,  # or another priority
       # ...
   }

2. Add required feed parameters:

   FEED_FILE

   absolute or relative file path where the result RSS feed will be saved. For example, feed.rss or output/feed.rss.

   FEED_TITLE

   the name of the channel (feed),

   FEED_DESCRIPTION

   phrase or sentence describing the channel (feed),

   FEED_LINK

   the URL to the HTML website corresponding to the channel (feed)

   FEED_FILE = 'path/to/feed.rss'
   FEED_TITLE = 'Some title of the channel'
   FEED_LINK = 'http://example.com/rss'
   FEED_DESCRIPTION = 'About channel'

Declare your item directly as RssItem():

import scrapy_rss

item1 = scrapy_rss.RssItem()

Or define new item class with RSS field named as rss:

import scrapy_rss

class MyItem(scrapy_rss.ExtendableItem):
    # scrapy.Field() and/or another fields definitions
    # ...

    def __init__(self):
        super(MyItem, self).__init__()
        self.rss = scrapy_rss.RssItem()

item2 = MyItem()

Set/get item fields. Case sensitive attributes of RssItem() are appropriate to RSS elements, Attributes of RSS elements are case sensitive too. If editor is allowed autocompletion then it suggests attributes for instances of RssItem. It’s allowed to set any subset of RSS elements (e.g. only title). For example:

from datetime import datetime

item1.title = 'RSS item title'  # set value of <title> element
title = item1.title.title  # get value of <title> element
item1.description = 'description'

item1.guid = 'item identifier'
item1.guid.isPermaLink = True  # set value of attribute isPermalink of <guid> element,
                               # isPermaLink is False by default
is_permalink = item1.guid.isPermaLink  # get value of attribute isPermalink of <guid> element
guid = item1.guid.guid  # get value of element <guid>

item1.category = 'single category'
category = item1.category
item1.category = ['first category', 'second category']
first_category = item1.category[0].category # get value of the element <category> with multiple values
all_categories = [cat.category for cat in item1.category]

# direct attributes setting
item1.enclosure.url = 'http://example.com/file'
item1.enclosure.length = 0
item1.enclosure.type = 'text/plain'

# or dict based attributes setting
item1.enclosure = {'url': 'http://example.com/file', 'length': 0, 'type': 'text/plain'}
item1.guid = {'guid': 'item identifier', 'isPermaLink': True}

item1.pubDate = datetime.now()  # correctly works with Python' datetimes


item2.rss.title = 'Item title'
item2.rss.guid = 'identifier'
item2.rss.enclosure = {'url': 'http://example.com/file', 'length': 0, 'type': 'text/plain'}

All allowed elements are listed in the scrapy_rss/items.py. All allowed attributes of each element with constraints and default values are listed in the scrapy_rss/elements.py.