Feedsearch Crawler
Feedsearch Crawler is a Python library for searching websites for RSS, Atom, and JSON feeds.
It is a continuation of my work on Feedsearch, which is itself a continuation of the work done by Dan Foreman-Mackey on Feedfinder2, which in turn is based on feedfinder - originally written by Mark Pilgrim and subsequently maintained by Aaron Swartz until his untimely death.
Feedsearch Crawler differs with all of the above in that it is now built as an asynchronous Web crawler for Python 3.7 and above, using asyncio and aiohttp, to allow much more rapid scanning of possible feed urls.
An implementation using this library to provide a public Feed Search API is available at https://feedsearch.auctorial.com
Installation
The library is available on PyPI:
pip install feedsearch-crawler
The library requires Python 3.7+.
Usage
Feedsearch Crawler is called with the single function search:
>>> from feedsearch_crawler import search
>>> feeds = search('xkcd.com')
>>> feeds
[FeedInfo('https://xkcd.com/rss.xml'), FeedInfo('https://xkcd.com/atom.xml')]
>>> feeds[0].url
URL('https://xkcd.com/rss.xml')
>>> str(feeds[0].url)
'https://xkcd.com/rss.xml'
>>> feeds[0].serialize()
{'url': 'https://xkcd.com/rss.xml', 'title': 'xkcd.com', 'version': 'rss20', 'score': 24, 'hubs': [], 'description': 'xkcd.com: A webcomic of romance and math humor.', 'is_push': False, 'self_url': '', 'favicon': 'https://xkcd.com/s/919f27.ico', 'content_type': 'text/xml; charset=UTF-8', 'bozo': 0, 'site_url': 'https://xkcd.com/', 'site_name': 'xkcd: Chernobyl', 'favicon_data_uri': '', 'content_length': 2847}
If you are already running in an asyncio event loop, then you can import and await search_async instead. The search function is only a wrapper that runs search_async in a new asyncio event loop.
from feedsearch_crawler import search_async
feeds = await search_async('xkcd.com')
A search will always return a list of FeedInfo objects, each of which will always have a url property, which is a URL object that can be decoded to a string with str(url).
The returned FeedInfo are sorted by the score value from highest to lowest, with a higher score theoretically indicating a more relevant feed compared to the original URL provided. A FeedInfo can also be serialized to a JSON compatible dictionary by calling it's .serialize() method.
The crawl logs can be accessed with:
import logging
logger = logging.getLogger("feedsearch_crawler")
Search Arguments
search and search_async take the following arguments:
search(
url: str,
try_urls: Union[List[str], bool]=False,
concurrency: int=10,
total_timeout: Union[float, aiohttp.ClientTimeout]=10,
request_timeout: Union[float, aiohttp.ClientTimeout]=3,
user_agent: str="Feedsearch Bot",
max_content_length: int=1024 * 1024 * 10,
max_depth: int=10,
headers: dict=None,
favicon_data_uri: bool=True
)
- url: str: The intial URL at which to search for feeds.
- try_urls: Union[List[str], bool]: An optional list of URL paths to query for feeds. Takes the origin of the url paramater and appends the provided paths. If no list is provided, but try_urls is True, then a list of common feed locations will be used.
- concurrency: int: An optional argument to specify the maximum number of concurrent HTTP requests.
- total_timeout: float: An optional argument to specify the time this function may run before timing out.
- request_timeout: float: An optional argument that controls how long before each individual HTTP request times out.
- user_agent: str: An optional argument to override the default User-Agent header.
- max_content_length: int: An optional argument to specify the maximum size in bytes of each HTTP Response.
- max_depth: int: An optional argument to limit the maximum depth of requests while following urls.
- headers: dict: An optional dictionary of headers to pass to each HTTP request.
- favicon_data_uri: bool: Optionally control whether to fetch found favicons and return them as a Data Uri.
FeedInfo Values
In addition to the url, FeedInfo objects may have the following values:
- bozo: int: Set to 1 when feed data is not well formed or may not be a feed. Defaults 0.
- content_length: int: Current length of the feed in bytes.
- content_type: str: Content-Type value of the returned feed.
- description: str: Feed description.
- favicon: URL: URL of feed or site Favicon.
- favicon_data_uri: str: Data Uri of Favicon.
- hubs: List[str]: List of Websub hubs of feed if available.
- is_push: bool: True if feed contains valid Websub data.
- score: int: Computed relevance of feed url value to provided URL. May be safely ignored.
- self_url: str: ref="self" value returned from feed links. In some cases may be different from feed url.
- site_name: str: Name of feed's website.
- site_url: URL: URL of feed's website.
- title: str: Feed Title.
- url: URL: URL location of feed.
- version: str: Feed version XML values, or JSON feed.
1 maintainer
Metadata
1 maintainer
Release files for feedsearch-crawler 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| feedsearch-crawler-0.1.1.tar.gz | 20.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| feedsearch_crawler-0.1.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 63.2 kB
Release files / feedsearch-crawler-0.1.1.tar.gz
| Download URL | feedsearch-crawler-0.1.1.tar.gz |
|---|---|
| Size | 20.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
088f07e5bbb0bfc64e3acbd2404b7ae1a106787d9dd2e320276e7c2d4014af30
|
|
BLAKE2b-256 checksum How to use checksums |
9754926637ce08e8aa9c18aed9f13c086dc79769a0a3613812a3fa12918e8bb5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.1 CPython/3.7.1
|
Release files / feedsearch_crawler-0.1.1-py3-none-any.whl
| Download URL | feedsearch_crawler-0.1.1-py3-none-any.whl |
|---|---|
| Size | 43.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8b811b37546cf62034ffbc6f8e2c546dd6278e1938554f852187ffe07a3e81af
|
|
BLAKE2b-256 checksum How to use checksums |
783ed589a18f939b5baecad64311dd3339e66b743edae138382cbebf492c9e73
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.1 CPython/3.7.1
|
