rssfixer 0.1.3

Generate RSS feed for Wordpress blog without it.

rssfixer

A small tool to generate an RSS feed from some WordPress blogs that for some reason don't generate their own feeds. This tool uses BeautifulSoup to parse the HTML and feedgen to generate the feed.

Installation

python3 -m venv venv
source venv/bin/activate
python3 -m pip install rssfixer

Example

Format for storing the links varies but I'll try and add more formats as I find them.

Simple list

An example to generate a feed for nccgroup that have the links in a simple ul-list:

$ rssfixer --title nccgroup https://research.nccgroup.com/
RSS feed created: rss_feed.xml

You can specify a filename and silence output:

rssfixer --title nccgroup --output nccgroup.xml --quiet https://research.nccgroup.com/

The resulting file is available here as an example.

Most times you would run the script from crontab to have an updated feed. Here is an example with a venv in /home/user/src/rssfixer.

32 * * * *      /home/user/src/rssfixer/bin/rssfixer --title nccgroup --output /var/www/html/feeds/nccgroup.xml --quiet https://research.nccgroup.com

JSON

Some blogs like truesec.com have the links in a JSON object. You can use the --json option to parse the JSON object and generate a feed. The same is true for Apple's security blog page.

An example for truesec.com:

rssfixer --json --quiet --output truesec.xml https://www.truesec.com/hub/blog

General HTML

Pages with a more general HTML structure can be parsed with the --html option. You can specify the HTML tag for the entries, the URL to the page and the title of the feed.

An example for tripwire.com:

rssfixer --title Tripwire --output tripwire.xml --quiet --html --base-url https://www.tripwire.com http://www.tripwire.com/state-of-security

Release

Check for one entity on release pages like SQLite (h3) and generate RSS feed with links to the download page (required argument --release-url). Easy way to get notified when a new version is released.

rssfixer --release --output sqlite.xml --release-entries h3 --release-url https://sqlite.org/download.html https://sqlite.org/changes.html

Usage

Command-line options:

usage: rssfixer [-h] (--html | --json | --list | --release) [--version]
                [--atom] [--base-url BASE_URL] [--release-url RELEASE_URL]
                [--release-entries RELEASE_ENTRIES]
                [--html-entries HTML_ENTRIES] [--html-url HTML_URL]
                [--html-title HTML_TITLE]
                [--html-title-class HTML_TITLE_CLASS]
                [--html-description HTML_DESCRIPTION]
                [--html-description-class HTML_DESCRIPTION_CLASS]
                [--json-entries JSON_ENTRIES] [--json-url JSON_URL]
                [--json-title JSON_TITLE]
                [--json-description JSON_DESCRIPTION] [--output OUTPUT]
                [--title TITLE] [-q] [--stdout]
                url

Generate RSS feed for blog that don't publish a feed. Default is to find links
in a simple <ul>-list. Options are available to find links in other HTML
elements or JSON strings.

positional arguments:
  url                   URL for the blog

options:
  -h, --help            show this help message and exit
  --html                Find entries in HTML
  --json                Find entries in JSON
  --list                Find entries in HTML <ul>-list (default)
  --release             Find releases in HTML
  --version             show program's version number and exit
  --atom                Generate Atom feed
  --base-url BASE_URL   Base URL for the blog
  --release-url RELEASE_URL
                        Release URL for downloads
  --release-entries RELEASE_ENTRIES
                        Release selector for entries
  --html-entries HTML_ENTRIES
                        HTML selector for entries
  --html-url HTML_URL   HTML selector for URL
  --html-title HTML_TITLE
                        HTML selector for title
  --html-title-class HTML_TITLE_CLASS
                        Flag to specify title class (regex)
  --html-description HTML_DESCRIPTION
                        HTML selector for description
  --html-description-class HTML_DESCRIPTION_CLASS
                        Flag to specify description class (regex)
  --json-entries JSON_ENTRIES
                        JSON key for entries (default: 'entries')
  --json-url JSON_URL   JSON key for URL (default: 'url')
  --json-title JSON_TITLE
                        JSON key for title
  --json-description JSON_DESCRIPTION
                        JSON key for description
  --output OUTPUT       Name of the output file
  --title TITLE         Title of the RSS feed (default: "My RSS Feed")
  -q, --quiet           Suppress output
  --stdout              Print to stdout

Command-line examples for some blogs

Apple Security Blog

Url: https://security.apple.com/blog/

rssfixer --title "Apple Security" --output apple.xml --quiet --json --json-entries blogs --json-url slug --base-url https://security.apple.com/blog/ https://security.apple.com/blog

nccgroup

Url: https://research.nccgroup.com/

rssfixer --title nccgroup --output nccgroup.xml --quiet --list https://research.nccgroup.com

Tripwire

Url: https://www.tripwire.com/state-of-security

rssfixer --title Tripwire --output tripwire.xml --quiet --html --base-url https://www.tripwire.com http://www.tripwire.com/state-of-security

TrueSec

Url: https://www.truesec.com/hub/blog

rssfixer --title Truesec --output truesec.xml --quiet --json --json-description preamble https://www.truesec.com/hub/blog

Sqlite

Url: https://sqlite.org/changes.html

rssfixer --release --release-entries h3 --release-url https://sqlite.org/download.html https://sqlite.org/changes.html

Setup blogs

During testing it is useful to use --stdout to see the generated feed.