RSS Tools for Scrapy Framework
Project description
Tools for easy RSS feed generating that contains each scraped item using Scrapy framework.
Package works with Python 2.7, 3.3, 3.4, 3.5 and 3.6.
If you use Python 3.3 then you have to use Scrapy<1.5.0.
Table of Contents
Installation
Install
scrapy_rss
using pippip install scrapy_rss
or using pip for the specific interpreter, e.g.:
pip3 install scrapy_rss
or using setuptools directly:
cd path/to/root/of/scrapy_rss python setup.py install
or using setuptools for specific interpreter, e.g.:
cd path/to/root/of/scrapy_rss python3 setup.py install
How To Use
Configuration
Add parameters to the Scrapy project settings (settings.py file)
or to the custom_settings
attribute of the spider:
Add item pipeline that export items to rss feed:
ITEM_PIPELINES = { # ... 'scrapy_rss.pipelines.RssExportPipeline': 900, # or another priority # ... }
Add required feed parameters:
- FEED_FILE
absolute or relative file path where the result RSS feed will be saved. For example,
feed.rss
oroutput/feed.rss
.- FEED_TITLE
the name of the channel (feed),
- FEED_DESCRIPTION
phrase or sentence describing the channel (feed),
- FEED_LINK
the URL to the HTML website corresponding to the channel (feed)
FEED_FILE = 'path/to/feed.rss' FEED_TITLE = 'Some title of the channel' FEED_LINK = 'http://example.com/rss' FEED_DESCRIPTION = 'About channel'
Feed (Channel) Elements Customization [optionally]
If you want to change another channel parameters (such as language, copyright, managing_editor,
webmaster, pubdate, last_build_date, category, generator, docs, ttl)
then declare your own exporter that’s inherited from RssItemExporter
class, for example:
from scrapy_rss.exporters import RssItemExporter
class MyRssItemExporter(RssItemExporter):
def __init__(self, *args, **kwargs):
kwargs['generator'] = kwargs.get('generator', 'Special generator')
kwargs['language'] = kwargs.get('language', 'en-us')
super(CustomRssItemExporter, self).__init__(*args, **kwargs)
And add FEED_EXPORTER
parameter to the Scrapy project settings
or to the custom_settings
attribute of the spider:
FEED_EXPORTER = 'myproject.exporters.MyRssItemExporter'
Usage
Declare your item directly as RssItem():
import scrapy_rss
item1 = scrapy_rss.RssItem()
Or use predefined item class RssedItem
with RSS field named as rss
that’s instance of RssItem
:
import scrapy_rss
class MyItem(scrapy_rss.RssedItem):
field1 = scrapy.Field()
field2 = scrapy.Field()
# ...
item2 = MyItem()
Set/get item fields. Case sensitive attributes of RssItem()
are appropriate to RSS elements,
Attributes of RSS elements are case sensitive too.
If editor allows autocomplete then it suggests attributes for instances of RssedItem
and RssItem
.
It’s allowed to set any subset of RSS elements (e.g. only title). For example:
from datetime import datetime
item1.title = 'RSS item title' # set value of <title> element
title = item1.title.title # get value of <title> element
item1.description = 'description'
item1.guid = 'item identifier'
item1.guid.isPermaLink = True # set value of attribute isPermalink of <guid> element,
# isPermaLink is False by default
is_permalink = item1.guid.isPermaLink # get value of attribute isPermalink of <guid> element
guid = item1.guid.guid # get value of element <guid>
item1.category = 'single category'
category = item1.category
item1.category = ['first category', 'second category']
first_category = item1.category[0].category # get value of the element <category> with multiple values
all_categories = [cat.category for cat in item1.category]
# direct attributes setting
item1.enclosure.url = 'http://example.com/file'
item1.enclosure.length = 0
item1.enclosure.type = 'text/plain'
# or dict based attributes setting
item1.enclosure = {'url': 'http://example.com/file', 'length': 0, 'type': 'text/plain'}
item1.guid = {'guid': 'item identifier', 'isPermaLink': True}
item1.pubDate = datetime.now() # correctly works with Python' datetimes
item2.rss.title = 'Item title'
item2.rss.guid = 'identifier'
item2.rss.enclosure = {'url': 'http://example.com/file', 'length': 0, 'type': 'text/plain'}
All allowed elements are listed in the scrapy_rss/items.py. All allowed attributes of each element with constraints and default values are listed in the scrapy_rss/elements.py. You also can read RSS specification for more details.
Scrapy Project Examples
Examples directory contains several Scrapy projects with the scrapy_rss usage demonstration. It crawls this website whose source code is here.
Just go to the Scrapy project directory and run commands
scrapy crawl first_spider
scrapy crawl second_spider
After this feed.rss and feed2.rss files will be created in the same directory.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for scrapy_rss-0.1.5-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 98bba4ee1a7d51bec662345bbbf7e4901a420318fe03807ae1e5683c4b14408b |
|
MD5 | fdc666eb50ec3c0d77415a89fb3676c6 |
|
BLAKE2b-256 | b2d691a3e13994f73557a3e355377a3112a0006bda23e2c727339364e585b7e5 |