Skip to main content
File added late

1 file was added to this release more than 14 days after its initial publication. Inspect the release files before installing.

jusText

https://api.travis-ci.org/miso-belica/jusText.png?branch=master

Program jusText is a tool for removing boilerplate content, such as navigation links, headers, and footers from HTML pages. It is designed to preserve mainly text containing full sentences and it is therefore well suited for creating linguistic resources such as Web corpora. You can try it online.

This is a fork of original (currently unmaintained) code of jusText hosted on Google Code.

Adaptations of the algorithm to other languages:

Some libraries using jusText:

Some currently (Jan 2020) maintained alternatives:

Installation

Make sure you have Python 2.7+/3.5+ and pip (Windows, Linux) installed. Run simply:

$ [sudo] pip install justext

Dependencies

lxml (version depends on your Python version)

Usage

$ python -m justext -s Czech -o text.txt http://www.zdrojak.cz/clanky/automaticke-zabezpeceni/
$ python -m justext -s English -o plain_text.txt english_page.html
$ python -m justext --help # for more info

Python API

import requests
import justext

response = requests.get("http://planet.python.org/")
paragraphs = justext.justext(response.content, justext.get_stoplist("English"))
for paragraph in paragraphs:
  if not paragraph.is_boilerplate:
    print paragraph.text

Testing

Run tests via

$ py.test-2.7 && py.test-3.5 && py.test-3.6 && py.test-3.7 && py.test-3.8 && py.test-3.9

Acknowledgements

This software has been developed at the Natural Language Processing Centre of Masaryk University in Brno with a financial support from PRESEMT and Lexical Computing Ltd. It also relates to PhD research of Jan Pomikálek.

Changelog for jusText

3.0.1 (2024-05-09)

  • BUG FIX: Fix issue with new version of lxml #48.

3.0.0 (2021-10-21)

  • INCOMPATIBLE CHANGE: Dropped support for Python 3.4 and below.

  • BUG FIX: Don’t join words separated only by <br> tag.

  • BUG FIX: List available stop-lists alphabetically.

2.2.0 (2016-03-06)

  • INCOMPATIBLE CHANGE: Stop words are case insensitive.

  • INCOMPATIBLE CHANGE: Dropped support for Python 3.2

  • BUG FIX: Preserve new lines from original text in paragraphs.

2.1.1 (2014-05-27)

  • BUG FIX: Function decode_html now respects parameter errors when falling to default_encoding #9.

2.1.0 (2014-01-25)

  • FEATURE: Added XPath selector to the paragrahs. XPath selector is also available in detailed output as xpath attribute of <p> tag #5.

2.0.0 (2013-08-26)

  • FEATURE: Added pluggable DOM preprocessor.

  • FEATURE: Added support for Python 3.2+.

  • INCOMPATIBLE CHANGE: Paragraphs are instances of justext.paragraph.Paragraph.

  • INCOMPATIBLE CHANGE: Script ‘justext’ removed in favour of command python -m justext.

  • FEATURE: It’s possible to enter an URI as input document in CLI.

  • FEATURE: It is possible to pass unicode string directly.

1.2.0 (2011-08-08)

  • FEATURE: Character counts used instead of word counts where possible in order to make the algorithm work well in the language independent mode (without a stoplist) for languages where counting words is not easy (Japanese, Chinese, Thai, etc).

  • BUG FIX: More robust parsing of meta tags containing the information about used charset.

  • BUG FIX: Corrected decoding of HTML entities &#128; to &#159;

1.1.0 (2011-03-09)

  • First public release.

Metadata

Release files for jusText 3.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

File added late

1 file was uploaded more than 14 days after the first file in this release.

While project maintainers occasionally add legitimate files to an existing release, late additions can also indicate a security compromise.

We recommend inspecting the release files before installing.

Source distribution (sdist)

Source distribution for jusText 3.0.1
File Size Uploaded Provenance Actions
justext-3.0.1.tar.gz 828.4 kB Without provenance Details for justext-3.0.1.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for jusText 3.0.1
File Interpreter ABI Platform Actions
justext-3.0.1-py2.py3-none-any.whl Python 2, Python 3 none any Details for justext-3.0.1-py2.py3-none-any.whl
jusText-3.0.1-py2.py3-none-any.whl Python 2, Python 3 none any Details for jusText-3.0.1-py2.py3-none-any.whl

Total release size: 2.5 MB

Release files / justext-3.0.1.tar.gz

Download URL justext-3.0.1.tar.gz
Size 828.4 kB
Tags Source
SHA-256 checksum
How to use checksums
b6ed2fb6c5d21618e2e34b2295c4edfc0bcece3bd549ed5c8ef5a8d20f0b3451
BLAKE2b-256 checksum
How to use checksums
b15993ce612fce25c274efc88ec4d65963ce80fce96b9048e9fc1e430d893a9e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.12.3

Release files / justext-3.0.1-py2.py3-none-any.whl

File added late

This file was uploaded more than 14 days after the first file in this release.

While project maintainers occasionally add legitimate files to an existing release, late additions can also indicate a security compromise.

We recommend inspecting the release file before installing.

Download URL justext-3.0.1-py2.py3-none-any.whl
Size 837.9 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
0a5225c5cd7c5a124fec7bfa9a55110a73135e8b58ce784470af67d051ac9fd3
BLAKE2b-256 checksum
How to use checksums
f4b2298d96f6b33fcb5ae7dbcbaea4e054601a575744de0e3edd98ae2a9d78e8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.2

Release files / jusText-3.0.1-py2.py3-none-any.whl

Download URL jusText-3.0.1-py2.py3-none-any.whl
Size 837.8 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
e0fb882dd7285415709f4b7466aed23d6b98b7b89404c36e8a2e730facfed02b
BLAKE2b-256 checksum
How to use checksums
c4302cd44d6cc7541d5a68848250bf2f12c588631f6ff4461421fee34f9b619e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.0.0 CPython/3.12.3

Security for jusText 3.0.1

Upload timing

File added late

1 file was uploaded more than 14 days after the first file in this release.

While project maintainers occasionally add legitimate files to an existing release, late additions can also indicate a security compromise.


Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page