Heuristic based boilerplate removal tool
Project description
jusText
jusText is a tool for removing boilerplate content, such as navigation links, headers, and footers from HTML pages. It is designed to preserve mainly text containing full sentences and it is therefore well suited for creating linguistic resources such as Web corpora. You can try it online.
This is the a fork of original code of jusText hosted on Google code. Below are some “forks” that I found on GitHub:
Instalation
Make sure you have Python installed.
Download the sources:
$ wget https://github.com/miso-belica/jusText/archive/master.zip
Extract the downloaded file:
$ unzip master.zip
Install the package (you may need sudo or a root shell for the latter command):
$ cd jusText-master/ $ python setup.py install
Or simply:
pip install git+git@github.com:miso-belica/jusText.git
Dependecies
lxml>=2.2.4
Usage
$ python -m justext -s english_page.html > plain_text.txt
$ python -m justext --help # for more info
Python API
import requests
import justext
response = requests.get('http://planet.python.org/')
paragraphs = justext.justext(response.content, justext.get_stoplist('English'))
for paragraph in paragraphs:
if paragraph['class'] == 'good':
print paragraph['text']
Acknowledgements
This software is developed at the Natural Language Processing Centre of Masaryk University in Brno with a financial support from PRESEMT and Lexical Computing Ltd. It also relates to PhD research of Jan Pomikalek.
Changelog for jusText
FEATURE: It is possible to pass unicode string directly.
1.2 (2011-08-08)
FEATURE: Character counts used instead of word counts where possible in order to make the algorithm work well in the language independent mode (without a stoplist) for languages where counting words is not easy (Japanese, Chinese, Thai, etc).
BUG FIX: More robust parsing of meta tags containing the information about used charset.
BUG FIX: Corrected decoding of HTML entities € to Ÿ
1.1 (2011-03-09)
First public release.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.