jusText

Program jusText is a tool for removing boilerplate content, such as navigation links, headers, and footers from HTML pages. It is designed to preserve mainly text containing full sentences and it is therefore well suited for creating linguistic resources such as Web corpora. You can try it online.

This is a fork of original (currently unmaintained) code of jusText hosted on Google Code. Below are some alternatives that I found:

Installation

Make sure you have Python 2.6+/3.2+ and pip (Windows, Linux) installed. Run simply (preferred way):

$ [sudo] pip install justext

Or for the fresh version:

$ [sudo] pip install git+git://github.com/miso-belica/jusText.git

Or if you have to:

$ wget https://github.com/miso-belica/jusText/archive/master.zip # download the sources
$ unzip master.zip # extract the downloaded file
$ jusText-master/
$ [sudo] python setup.py install # install the package

Dependencies

lxml>=2.2.4

Usage

$ python -m justext -s Czech -o text.txt http://www.zdrojak.cz/clanky/automaticke-zabezpeceni/
$ python -m justext -s English -o plain_text.txt english_page.html
$ python -m justext --help # for more info

Python API

import requests
import justext

response = requests.get("http://planet.python.org/")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
  if not paragraph.is_boilerplate:
    print paragraph.text

Testing

Run tests via

$ nosetests-2.6 && nosetests-3.2 && nosetests-2.7 && nosetests-3.3

Acknowledgements

This software has been developed at the Natural Language Processing Centre of Masaryk University in Brno with a financial support from PRESEMT and Lexical Computing Ltd. It also relates to PhD research of Jan Pomikálek.

Metadata

Release files for jusText 2.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for jusText 2.1.1
File	Size	Uploaded
jusText-2.1.1.zip	864.5 kB	May 27, 2014	Details

Built distribution (wheel)

Table of built distributions (wheels) for jusText 2.1.1
File	Interpreter	ABI	Platform	Reset
jusText-2.1.1-py2.py3-none-any.whl 861.1 kB May 27, 2014	Python 2, Python 3	none	any	Details

Total release size: 1.7 MB

Release files / jusText-2.1.1.zip

Download URL	jusText-2.1.1.zip
Size	864.5 kB
Tags	Source
SHA-256 checksum How to use checksums	`1dd07067f415a87f22323a54d3d2238c9b0fe8e7945551a0763496ac56d30a16`
BLAKE2b-256 checksum How to use checksums	`046d31e00efd44497ffa0e433d5f8eca61216ad3feba4ee7b0dcb76ba9d75df4`
Upload date	May 27, 2014
Uploaded using Trusted Publishing? What is trusted publishing?	No

Release files / jusText-2.1.1-py2.py3-none-any.whl

Download URL	jusText-2.1.1-py2.py3-none-any.whl
Size	861.1 kB
Tags	Python 2 Python 3
SHA-256 checksum How to use checksums	`5361f7d40568d6bdf2432aeba0bc11d1107db565e712bdb6624eec1f98e594ee`
BLAKE2b-256 checksum How to use checksums	`151ea97c2cd1169e87e74e954e9b3946c6fc6b34e5178268d1cbabd6209ae289`
Upload date	May 27, 2014
Uploaded using Trusted Publishing? What is trusted publishing?	No