Skip to main content

Sitemap scraper for news article selection within a certain time range

Project description

About

This module provides the SitemapRange class and a tool to allow command-line usage sitemap_fetch.py.

The class SitemapRange is meant primarily as a generic building block for creating news aggregating applications where the datasources are spec-compliant news websites.

There are some fault-tolerance features included to deal with some inconsistencies in sitemaps.

Install

To install from pypi:

pip3 install --user sitemap-range-fetch

Usage

Fetching all news articles on cnn.com in the past 6 days, and format the result as JSON:

sitemap_fetch.py --site "https://cnn.com" --format json --daysago 6

Here is an example of using the SitemapRange class in your code:

from sitemap_range.sitemap_range import SitemapRange
from datetime import datetime, timedelta
sr = SitemapRange("https://cnn.com")
in_range = sr.get_articles_in_range(start=datetime.now()-timedelta(days=3), end=datetime.now(), opts={})
print(in_range)

The get_articles_in_range method returns a list of dictionaries, where each dictionary has two keys: "url" and "dt" which is an ISO 8601 formatted datetime string (as returned by the isoformat method).

More details about the CLI switches:

    usage: sitemap_fetch.py [-h] --site SITE [--format FORMAT] [--daysago DAYSAGO]
                            [--notz] [--advanced]

    Tool for extracting articles from news websites

    optional arguments:
      -h, --help         show this help message and exit
      --site SITE        the url for the website
      --format FORMAT    the url for the website
      --daysago DAYSAGO  defines the oldest date of an article that will be
                         selected (default: 2 days ago)
      --notz             strip the timezone from the dates before selection
                         (processing is more fault-tolerant)
      --advanced         use a more fault-tolerant parser

Details

This module is provided as is under MIT License.

For extensions, customizations or business inquiries you can get in touch here.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sitemap-range-fetch-0.9.5.tar.gz (4.8 kB view hashes)

Uploaded Source

Built Distribution

sitemap_range_fetch-0.9.5-py3-none-any.whl (6.5 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page