Skip to main content

Extract text from HTML

Project description

HTML to Text

PyPI Version Build Status Code Coverage

Extract text from HTML

  • Free software: MIT license

How is html_text different from .xpath('//text()') from LXML or .get_text() from Beautiful Soup? Text extracted with html_text does not contain inline styles, javascript, comments and other text that is not normally visible to the users. It normalizes whitespace, but is also smarter than .xpath('normalize-space()), adding spaces around inline elements too (which are often used as block elements in html markup), and tries to avoid adding extra spaces for punctuation.

Apart from just getting text from the page (e.g. for display or search), one intended usage of this library is for machine learning (feature extraction). If you want to use the text of the html page as a feature (e.g. for classification), this library gives you plain text that you can later feed into a standard text classification pipeline. If you feel that you need html structure as well, check out webstruct library.

Install

Install with pip:

pip install html-text

The package depends on lxml, so you might need to install some additional packages: http://lxml.de/installation.html

Usage

Extract text from HTML:

>>> import html_text
>>> text = html_text.extract_text(u'<h1>Hey</h1>')
u'Hey'

You can also pass already parsed lxml.html.HtmlElement:

>>> import html_text
>>> tree = html_text.parse_html(u'<h1>Hey</h1>')
>>> text = html_text.extract_text(tree)
u'Hey'

Passed html will be first cleaned from invisible non-text content such as styles, and then text would be extracted. Two functions that do it are html_text.cleaned_selector and html_text.selector_to_text:

  • html_text.cleaned_selector accepts html as text or as lxml.html.HtmlElement, and returns cleaned parsel.Selector.
  • html_text.selector_to_text accepts parsel.Selector and returns extracted text.

Credits

The code is extracted from utilities used in several projects, written by Mikhail Korobov.

History

0.3.0 (2017-10-12)

  • Expose functions that operate on selectors, use .//text() to extract text from selector.

0.2.1 (2017-05-29)

  • Packaging fix (include CHANGES.rst)

0.2.0 (2017-05-29)

  • Fix unwanted joins of words with inline tags: spaces are added for inline tags too, but a heuristic is used to preserve punctuation without extra spaces.
  • Accept parsed html trees.

0.1.1 (2017-01-16)

  • Travis-CI and codecov.io integrations added

0.1.0 (2016-09-27)

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for html-text, version 0.3.0
Filename, size File type Python version Upload date Hashes
Filename, size html_text-0.3.0.tar.gz (5.7 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page