Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Gathers unstructured News data into a SQLite3 db

Project Description

Gathers unstructured news data and commits it to a SQLite3 database.The goal for GatherNews is to quickly and simply capture news data.

GatherNews allows you to specify which News sites you want to capture by providing the RSS link in “feeds_list.txt” like this:

http://feeds.reuters.com/Reuters/worldNews
http://rss.cnn.com/rss/money_latest.rss

You can then gather the news using the load_db() method:

>>> # Create new tables if any new RSS feed addresses have been added
>>> # Populate all tables with RSS news feeds
>>> # Remove duplicate entries
>>> capture_feeds.load_db()

The examples folder contains working code for each module.

Features

  • Creates tables with a predefined schema
  • Populates each table in the SQLite3 database with articles
  • Removes duplicate articles from each table
  • Includes bug fixes for issues raised about version 0.1.0
  • Faster than version 0.1.0, see the benchmarks!

Installation

To install GatherNews use pip:

$ pip install gathernews

Testing

After installation, you can launch the test suite from outside the source directory (you will need to have nosetests installed):

$ nosetests --exe GatherNews

Bug Resolution

In GatherNews 0.1.0, a bug was introduced that does not allow you to add new RSS feeds to the ‘feeds_list.txt’ after your initial call of the create_tables() method. Before using version 0.2.0 on a preexisting database call the following method.

>>> path = "path/to_your/existing_sqlite3_db/
>>> from gathernews.gRSS import CaptureFeeds
>>> capture_feeds = CaptureFeeds(path)
>>> capture_feeds.fix_create_table_bug()

If this gives you any problems, email me directly (tylers.pile@gmail.com). Thanks.

Documentation

Documentation is available at http://gathernews.readthedocs.org/en/latest/

Contribute

  1. Issue tracker is here: https://github.com/Bonza-Times/GatherNews/issues
  2. Fork it!
Release History

Release History

This version
History Node

0.2.1

History Node

0.2.0

History Node

0.1.1

History Node

0.1.0

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
GatherNews-0.2.1.tar.gz (10.1 kB) Copy SHA256 Checksum SHA256 Source Feb 16, 2014

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting