Skip to main content
Join the official Python Developers Survey 2018 and win valuable prizes: Start the survey!

Scrape mailman archive and convert to rss

Project description

A simple mailman archive -> rss converter. Also the command can run as a twitter bot.

Install

Install the package with pip.

pip install mailman-rss

Twitter bot can be installed with an option:

pip install mailman-rss[twitter]

Usage

usage: mailman-rss [-h] [--command COMMAND] [--config CONFIG]
                   [--archive-url URL] [--max-items MAX_ITEMS] [-o OUTPUT]
                   [-l LOGLEVEL] [--encoding ENCODING] [--language LANGUAGE]

Fetch Mailman archive and export to RSS

optional arguments:
  -h, --help            show this help message and exit
  --command COMMAND     Command to execute, rss or twitter
  --config CONFIG       mailman-rss config json file, default None
  --archive-url URL     Archive URL to fetch
  --max-items MAX_ITEMS
                        number of messages to convert to rss
  -o OUTPUT, --output OUTPUT
                        output file name, default stdout
  -l LOGLEVEL, --loglevel LOGLEVEL
                        logging level for debugging, default warning
  --encoding ENCODING   email message encoding, default None
  --language LANGUAGE   language specification, default None

RSS Example: this will output RSS feed to stdout.

mailman-rss --archive-url http://example.com/mailman/list/

The command can have a config file.

mailman-rss --config /path/to/config.json
{
  "archive_url": "http://example.com/mailman/list/",
  "output": "/home/user/public_html/mailman.rss"
}

Twitter example: twitter bot requires an application token and a storage space. Get Twitter API Key to run the command.

{
  "command": "twitter",
  "archive_url": "http://example.com/mailman/list/",
  "db": "/home/user/scraper.sqlite3",
  "max_items": 10,
  "consumer_key": "XXXXXXXXXXXXXXXXXXXXXXXXXX",
  "consumer_secret": "XXXXXXXXXXXXXXXXXXXXXXXXXX",
  "access_token_key": "XXXXXXX-XXXXXXXXXXXXXXXXXXXXXXXXXX",
  "access_token_secret": "XXXXXXXXXXXXXXXXXXXXXXXXXX"
}

Deploy

The command should be run from cron.

crontab -l

0 * * * * mailman-rss --archive-url http://example.com/mailman/list/ > /var/www/archive.rss
0 * * * * mailman-rss --config /home/user/twitter-bot.json

Test

tox

Project details


Release history Release notifications

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
mailman_rss-0.2.0-py3-none-any.whl (11.1 kB) Copy SHA256 hash SHA256 Wheel py3 Dec 26, 2017

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page