This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Heuristic based boilerplate removal tool

Project Description

jusText

Program jusText is a tool for removing boilerplate content, such as navigation links, headers, and footers from HTML pages. It is designed to preserve mainly text containing full sentences and it is therefore well suited for creating linguistic resources such as Web corpora. You can try it online.

This is a fork of original (currently unmaintained) code of jusText hosted on Google Code. Below are some alternatives that I found:

Installation

Make sure you have Python 2.6+/3.3+ and pip (Windows, Linux) installed. Run simply:

$ [sudo] pip install justext

Dependencies

lxml>=2.2.4

Usage

$ python -m justext -s Czech -o text.txt http://www.zdrojak.cz/clanky/automaticke-zabezpeceni/
$ python -m justext -s English -o plain_text.txt english_page.html
$ python -m justext --help # for more info

Python API

import requests
import justext

response = requests.get("http://planet.python.org/")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
  if not paragraph.is_boilerplate:
    print paragraph.text

Testing

Run tests via

$ py.test-2.6 && py.test-3.3 && py.test-2.7 && py.test-3.4 && py.test-3.5

Acknowledgements

This software has been developed at the Natural Language Processing Centre of Masaryk University in Brno with a financial support from PRESEMT and Lexical Computing Ltd. It also relates to PhD research of Jan Pomikálek.

Changelog for jusText

2.2.0 (2016-03-06)

  • INCOMPATIBLE CHANGE: Stop words are case insensitive.
  • INCOMPATIBLE CHANGE: Dropped support for Python 3.2
  • BUG FIX: Preserve new lines from original text in paragraphs.

2.1.1 (2014-05-27)

  • BUG FIX: Function decode_html now respects parameter errors when falling to default_encoding #9.

2.1.0 (2014-01-25)

  • FEATURE: Added XPath selector to the paragrahs. XPath selector is also available in detailed output as xpath attribute of <p> tag #5.

2.0.0 (2013-08-26)

  • FEATURE: Added pluggable DOM preprocessor.
  • FEATURE: Added support for Python 3.2+.
  • INCOMPATIBLE CHANGE: Paragraphs are instances of justext.paragraph.Paragraph.
  • INCOMPATIBLE CHANGE: Script ‘justext’ removed in favour of command python -m justext.
  • FEATURE: It’s possible to enter an URI as input document in CLI.
  • FEATURE: It is possible to pass unicode string directly.

1.2.0 (2011-08-08)

  • FEATURE: Character counts used instead of word counts where possible in order to make the algorithm work well in the language independent mode (without a stoplist) for languages where counting words is not easy (Japanese, Chinese, Thai, etc).
  • BUG FIX: More robust parsing of meta tags containing the information about used charset.
  • BUG FIX: Corrected decoding of HTML entities &#128; to &#159;

1.1.0 (2011-03-09)

  • First public release.
Release History

Release History

This version
History Node

2.2.0

History Node

2.1.1

History Node

2.1.0

History Node

2.0.0

History Node

1.2.0

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
jusText-2.2.0-py2.py3-none-any.whl (860.8 kB) Copy SHA256 Checksum SHA256 py2.py3 Wheel Mar 6, 2016
jusText-2.2.0.zip (864.1 kB) Copy SHA256 Checksum SHA256 Source Mar 6, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting