RSS Tools for Scrapy Framework
Project description
Tools for easy generating of RSS feed with each scraped item using Scrapy framework.
Package works with Python 2.7, 3.3, 3.4 and 3.5.
Installation
Install
scrapy_rss
using pippip install scrapy_rss
or using pip for specific interpreter, e.g.:
pip3 install scrapy_rss
or using directly setuptools:
cd path/to/root/of/scrapy_rss python setup.py install
or using setuptools for specific interpreter, e.g.:
cd path/to/root/of/scrapy_rss python3 setup.py install
How To Use
Add parameters to the Scrapy project settings (settings.py file)
or to the custom_settings
attribute of the spider:
Add item pipeline that export items to rss feed:
ITEM_PIPELINES = { # ... 'scrapy_rss.pipelines.RssExportPipeline': 900, # or another priority # ... }
Add required feed parameters:
- FEED_FILE
absolute or relative file path where the result RSS feed will be saved. For example,
feed.rss
oroutput/feed.rss
.- FEED_TITLE
the name of the channel (feed),
- FEED_DESCRIPTION
phrase or sentence describing the channel (feed),
- FEED_LINK
the URL to the HTML website corresponding to the channel (feed)
FEED_FILE = 'path/to/feed.rss' FEED_TITLE = 'Some title of the channel' FEED_LINK = 'http://example.com/rss' FEED_DESCRIPTION = 'About channel'
Declare your item directly as RssItem():
import scrapy_rss
item1 = scrapy_rss.RssItem()
Or define new item class with RSS field named as rss
:
import scrapy_rss
class MyItem(scrapy_rss.ExtendableItem):
# scrapy.Field() and/or another fields definitions
# ...
def __init__(self):
super(MyItem, self).__init__()
self.rss = scrapy_rss.RssItem()
item2 = MyItem()
Set/get item fields. Case sensitive attributes of RssItem()
are appropriate to RSS elements,
Attributes of RSS elements are case sensitive too.
If editor is allowed autocompletion then it suggests attributes for instances of RssItem
.
It’s allowed to set any subset of RSS elements (e.g. only title). For example:
item1.title = 'RSS item title' # set value of <title> element
title = item1.title.title # get value of <title> element
item1.description = 'description'
item1.guid = 'item identifier'
item1.guid.isPermaLink = True # set value of attribute isPermalink of <guid> element,
# isPermaLink is False by default
is_permalink = item1.guid.isPermaLink # get value of attribute isPermalink of <guid> element
guid = item1.guid.guid # set guid to 'item identifier'
item1.category = 'single category'
category = item1.category
item1.category = ['first category', 'second category']
first_category = item1.category[0].category # get value of the element <category> with multiple values
all_categories = [cat.category for cat in item1.category]
# direct attributes setting
item1.enclosure.url = 'http://example.com/file'
item1.enclosure.length = 0
item1.enclosure.type = 'text/plain'
# or dict based attributes setting
item1.enclosure = {'url': 'http://example.com/file', 'length': 0, 'type': 'text/plain'}
item1.guid = {'guid': 'item identifier', 'isPermaLink': True}
item2.rss.title = 'Item title'
item2.rss.guid = 'identifier'
item2.rss.enclosure = {'url': 'http://example.com/file', 'length': 0, 'type': 'text/plain'}
All allowed elements are listed in the scrapy_rss/items.py. All allowed attributes of each element with constraints and default values are listed in the scrapy_rss/elements.py.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for scrapy_rss-0.1.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9fa39f687481ed74f141f493ca371a3d28f9531df652746dace3cba7b8c7ae5f |
|
MD5 | 90d01588be0fcbfa6a94c0cae7e86b90 |
|
BLAKE2b-256 | 45660d9898555b12030992e82c543031c3239f8bca88105558c6322587eec1b2 |