Skip to main content

Modified arc90's scraping hub port readability tool for dig data

Project description

This code is under the Apache License 2.0. http://www.apache.org/licenses/LICENSE-2.0

This is a modfied for higher recall rate of famous python port(https://github.com/buriy/python-readability) a ruby port of arc90’s readability project

http://lab.arc90.com/experiments/readability/ https://github.com/buriy/python-readability

In few words, Given a html document, it pulls out the main body text and cleans it up. It also can clean up title based on latest readability.js code.

Based on:

Installation:

pip install readability-dig

Usage:

from readability.readability import Document
import urllib
html = urllib.urlopen(url).read()
readable_article = Document(html).summary()
readable_title = Document(html).short_title()

Command-line usage:

python -m readability.readability -u http://pypi.python.org/pypi/readability-dig

To open resulting page in browser:

python -m readability.readability -b -u http://pypi.python.org/pypi/readability-dig

Using positive/negative keywords example:

python -m readability.readability -p intro -n newsindex,homepage-box,news-section -u http://python.org

Document() kwarg options:

  • attributes:
  • debug: output debug messages
  • min_text_length:
  • retry_length:
  • url: will allow adjusting links to be absolute
  • positive_keywords: the list of positive search patterns in classes and ids, for example: [“news-item”, “block”]
  • negative_keywords: the list of negative search patterns in classes and ids, for example: [“mysidebar”, “related”, “ads”]

Project details


Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page