Skip to main content

Feedsearch Crawler

PyPI PyPI - Python Version PyPI - License Downloads

Feedsearch Crawler is a Python library for searching websites for RSS, Atom, and JSON feeds.

It is a continuation of my work on Feedsearch, which is itself a continuation of the work done by Dan Foreman-Mackey on Feedfinder2, which in turn is based on feedfinder - originally written by Mark Pilgrim and subsequently maintained by Aaron Swartz until his untimely death.

Feedsearch Crawler differs with all of the above in that it is now built as an asynchronous Web crawler for Python 3.7 and above, using asyncio and aiohttp, to allow much more rapid scanning of possible feed urls.

An implementation using this library to provide a public Feed Search API is available at https://feedsearch.auctorial.com

Installation

The library is available on PyPI:

pip install feedsearch-crawler

The library requires Python 3.7+.

Usage

Feedsearch Crawler is called with the single function search:

>>> from feedsearch_crawler import search
>>> feeds = search('xkcd.com')
>>> feeds
[FeedInfo('https://xkcd.com/rss.xml'), FeedInfo('https://xkcd.com/atom.xml')]
>>> feeds[0].url
URL('https://xkcd.com/rss.xml')
>>> str(feeds[0].url)
'https://xkcd.com/rss.xml'
>>> feeds[0].serialize()
{'url': 'https://xkcd.com/rss.xml', 'title': 'xkcd.com', 'version': 'rss20', 'score': 24, 'hubs': [], 'description': 'xkcd.com: A webcomic of romance and math humor.', 'is_push': False, 'self_url': '', 'favicon': 'https://xkcd.com/s/919f27.ico', 'content_type': 'text/xml; charset=UTF-8', 'bozo': 0, 'site_url': 'https://xkcd.com/', 'site_name': 'xkcd: Chernobyl', 'favicon_data_uri': '', 'content_length': 2847}

If you are already running in an asyncio event loop, then you can import and await search_async instead. The search function is only a wrapper that runs search_async in a new asyncio event loop.

from feedsearch_crawler import search_async

feeds = await search_async('xkcd.com')

A search will always return a list of FeedInfo objects, each of which will always have a url property, which is a URL object that can be decoded to a string with str(url). The returned FeedInfo are sorted by the score value from highest to lowest, with a higher score theoretically indicating a more relevant feed compared to the original URL provided. A FeedInfo can also be serialized to a JSON compatible dictionary by calling it's .serialize() method.

The crawl logs can be accessed with:

import logging

logger = logging.getLogger("feedsearch_crawler")

Feedsearch Crawler also provides a handy function to output the returned feeds as an OPML subscription list, encoded as a UTF-8 bytestring.

from feedsearch_crawler import output_opml

output_opml(feeds).decode()

Search Arguments

search and search_async take the following arguments:

search(
    url: str,
    try_urls: Union[List[str], bool]=False,
    concurrency: int=10,
    total_timeout: Union[float, aiohttp.ClientTimeout]=10,
    request_timeout: Union[float, aiohttp.ClientTimeout]=3,
    user_agent: str="Feedsearch Bot",
    max_content_length: int=1024 * 1024 * 10,
    max_depth: int=10,
    headers: dict={"X-Custom-Header": "Custom Header"},
    favicon_data_uri: bool=True,
    delay: float=0
)
  • url: str: The initial URL at which to search for feeds.
  • crawl_hosts: bool: (default False): An optional argument to add the site host origin URL (e.g. http://example.com) to the list of initial crawl URLs.
  • try_urls: Union[List[str], bool]: (default False): An optional list of URL paths to query for feeds. Takes the origin of the url paramater and appends the provided paths. If no list is provided, but try_urls is True, then a list of common feed locations will be used.
  • concurrency: int: (default 10): An optional argument to specify the maximum number of concurrent HTTP requests.
  • total_timeout: float: (default 30.0): An optional argument to specify the time this function may run before timing out.
  • request_timeout: float: (default 3.0): An optional argument that controls how long before each individual HTTP request times out.
  • user_agent: str: An optional argument to override the default User-Agent header.
  • max_content_length: int: (default 10Mb): An optional argument to specify the maximum size in bytes of each HTTP Response.
  • max_depth: int: (default 10): An optional argument to limit the maximum depth of requests while following urls.
  • headers: dict: An optional dictionary of headers to pass to each HTTP request.
  • favicon_data_uri: bool: (default True): Optionally control whether to fetch found favicons and return them as a Data Uri.
  • delay: float: (default 0.0): An optional argument to delay each HTTP request by the specified time in seconds. Used in conjunction with the concurrency setting to avoid overloading sites.

FeedInfo Values

In addition to the url, FeedInfo objects may have the following values:

  • bozo: int: Set to 1 when feed data is not well formed or may not be a feed. Defaults 0.
  • content_length: int: Current length of the feed in bytes.
  • content_type: str: Content-Type value of the returned feed.
  • description: str: Feed description.
  • favicon: URL: URL of feed or site Favicon.
  • favicon_data_uri: str: Data Uri of Favicon.
  • hubs: List[str]: List of Websub hubs of feed if available.
  • is_podcast: bool: True if the feed contains valid podcast elements and enclosures.
  • is_push: bool: True if feed contains valid Websub data.
  • last_updated: datetime: Date of the latest published entry.
  • score: int: Computed relevance of feed url value to provided URL. May be safely ignored.
  • self_url: str: ref="self" value returned from feed links. In some cases may be different from feed url.
  • site_name: str: Name of feed's website.
  • site_url: URL: URL of feed's website.
  • title: str: Feed Title.
  • url: URL: URL location of feed.
  • velocity: float: Mean number of entries per day in the feed at the current time.
  • version: str: Feed version XML values, or JSON feed.

Metadata

Release files for feedsearch-crawler 0.1.20

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for feedsearch-crawler 0.1.20
File Size Uploaded Provenance Actions
feedsearch-crawler-0.1.20.tar.gz 32.3 kB Without provenance Details for feedsearch-crawler-0.1.20.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for feedsearch-crawler 0.1.20
File Interpreter ABI Platform Actions
feedsearch_crawler-0.1.20-py3-none-any.whl Python 3 none any Details for feedsearch_crawler-0.1.20-py3-none-any.whl

Total release size: 70.9 kB

Release files / feedsearch-crawler-0.1.20.tar.gz

Download URL feedsearch-crawler-0.1.20.tar.gz
Size 32.3 kB
Tags Source
SHA-256 checksum
How to use checksums
9f62e6d5302d5bda4f901ad5869a05b30b764e70cabcd7fce67a8a2d5cb36359
BLAKE2b-256 checksum
How to use checksums
158bc32af96e1aa507704b2821d201c34eaa95789dc67545d6e29d4304147abf
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.1

Release files / feedsearch_crawler-0.1.20-py3-none-any.whl

Download URL feedsearch_crawler-0.1.20-py3-none-any.whl
Size 38.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
8a54b2b5f1bc1199b3549e0eac50dab5fc797564738ce2ae70ff01a197d325c8
BLAKE2b-256 checksum
How to use checksums
748ec4bd3d1e0ddd4ac0cf66b96da6c60aba62f4237d1b6e43288f192beabace
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.15.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.36.1 CPython/3.7.1

Security for feedsearch-crawler 0.1.20

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.20 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page