Installation

Make sure you have Python 2.7+/3.5+ and pip (Windows, Linux) installed. Run simply:

$ [sudo] pip install justext

Dependencies

lxml (version depends on your Python version)

Usage

$ python -m justext -s Czech -o text.txt http://www.zdrojak.cz/clanky/automaticke-zabezpeceni/
$ python -m justext -s English -o plain_text.txt english_page.html
$ python -m justext --help # for more info

Python API

import requests
import justext

response = requests.get("http://planet.python.org/")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
  if not paragraph.is_boilerplate:
    print paragraph.text

Testing

Run tests via

$ py.test-2.7 && py.test-3.5 && py.test-3.6 && py.test-3.7 && py.test-3.8 && py.test-3.9

Acknowledgements

This software has been developed at the Natural Language Processing Centre of Masaryk University in Brno with a financial support from PRESEMT and Lexical Computing Ltd. It also relates to PhD research of Jan Pomikálek.

2.2.0 (2016-03-06)

• INCOMPATIBLE CHANGE: Stop words are case insensitive.

• INCOMPATIBLE CHANGE: Dropped support for Python 3.2

• BUG FIX: Preserve new lines from original text in paragraphs.

2.1.1 (2014-05-27)

• BUG FIX: Function decode_html now respects parameter errors when falling to default_encoding #9.

2.1.0 (2014-01-25)

• FEATURE: Added XPath selector to the paragrahs. XPath selector is also available in detailed output as xpath attribute of <p> tag #5.

2.0.0 (2013-08-26)

• FEATURE: Added pluggable DOM preprocessor.

• FEATURE: Added support for Python 3.2+.

• INCOMPATIBLE CHANGE: Paragraphs are instances of justext.paragraph.Paragraph.

• INCOMPATIBLE CHANGE: Script ‘justext’ removed in favour of command python -m justext.

• FEATURE: It’s possible to enter an URI as input document in CLI.

• FEATURE: It is possible to pass unicode string directly.

1.2.0 (2011-08-08)

• FEATURE: Character counts used instead of word counts where possible in order to make the algorithm work well in the language independent mode (without a stoplist) for languages where counting words is not easy (Japanese, Chinese, Thai, etc).

• BUG FIX: More robust parsing of meta tags containing the information about used charset.

• BUG FIX: Corrected decoding of HTML entities &#128; to &#159;

1.1.0 (2011-03-09)

• First public release.