inscriptis - HTML to text converter.
Project description
inscriptis
A python based HTML to text conversion library, command line client and Web service with support for nested tables and a subset of CSS. Please take a look at the Rendering document for a demonstration of inscriptis' conversion quality.
Table of Contents
- Requirements and installation
- Command line client
- Python library
- Web service
- Fine tuning
- Testing, benchmarking and evaluation
- Changelog
Requirements and installation
Requirements
- Python 3.5+ (preferred) or Python 2.7+
- lxml
- requests
Installation
sudo python3 setup.py install
Command line client
The command line client converts text files or text retrieved from Web pages to the corresponding text representation.
Command line parameters
usage: inscript.py [-h] [-o OUTPUT] [-e ENCODING] [-i] [-l] [-d] input
Converts HTML from file or url to a clean text version
positional arguments:
input Html input either from a file or an url (default:stdin)
optional arguments:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
Output file (default:stdout).
-e ENCODING, --encoding ENCODING
Content encoding for files (default:utf-8)
-i, --display-image-captions
Display image captions (default:false).
-l, --display-link-targets
Display link targets (default:false).
-d, --deduplicate-image-captions
Deduplicate image captions (default:false).
--indentation
How to handle indentation (extended or standard; default: extended)
Examples
# convert the given page to text and output the result to the screen
inscript.py https://www.fhgr.ch
# convert the file to text and save the output to output.txt
inscript.py fhgr.html -o fhgr.txt
# convert the text provided via stdin and save the output to output.txt
echo '<body><p>Make it so!</p>></body>' | inscript.py -o output.txt
Python library
Embedding inscriptis into your code is easy, as outlined below:
import urllib.request
from inscriptis import get_text
url = "http://www.informationscience.ch"
html = urllib.request.urlopen(url).read().decode('utf-8')
text = get_text(html)
print(text)
Flask Web Service
The Flask Web Service translates HTML pages to the corresponding plain text.
Additional Requirements
- python3-flask
Startup
export FLASK_APP="web-service.py"
python3 -m flask run
Usage
The Web services receives the HTML file in the request body and returns the corresponding text. The file's encoding needs to be specified
in the Content-Type header (UTF-8 in the example below).
curl -X POST -H "Content-Type: text/html; encoding=UTF8" -d @test.html http://localhost:5000/get_text
Fine tuning
The following options are available for fine tuning the way inscriptis translates HTML to text.
-
More rigorous indentation: call
inscriptis.get_text()with the parameterindentation='extended'to also use indentation for tags such as<div>and<span>that do not provide indentation in their standard definition. This strategy is the default ininscript.pyand many other tools such as lynx. If you do not want extended indentation you can use the parameterindentation='standard'instead. -
Overwriting the default CSS definition: inscriptis uses CSS definitions that are maintained in
inscriptis.css.CSSfor rendering HTML tags. You can override these definitions (and therefore change the rendering) as outlined below:from lxml.html import fromstring from inscriptis.css import DEFAULT_CSS, HtmlElement from inscriptis.html_properties import Display # create a custom CSS based on the default style sheet and change the rendering of `div` and `span` elements css = DEFAULT_CSS.copy() css['div'] = HtmlElement('div', display=Display.block, padding=2) css['span'] = HtmlElement('span', prefix=' ', suffix=' ') html_tree = fromstring(html) # create a parser using the custom css parser = Inscriptis(html_tree, display_images=display_images, deduplicate_captions=deduplicate_captions, display_links=display_links, css=css) text = parser.get_text()
Testing, benchmarking and evaluation
Unit tests
Test cases concerning the html to text conversion are located in the tests/html directory and consist of two files:
test-name.htmlandtest-name.txt
the latter one containing the reference text output for the given html file.
Text conversion output comparison and speed benchmarking
inscriptis offers a small benchmarking script that can compare different HTML to text conversion approaches.
The script will run the different approaches on a list of URLs, url_list.txt, and save the text output into a time stamped folder in benchmarking/benchmarking_results for manual comparison.
Additionally the processing speed of every approach per URL is measured and saved in a text file called speed_comparisons.txt in the respective time stamped folder.
To run the benchmarking script execute run_benchmarking.py from within the folder benchmarking.
In def pipeline() set the which HTML -> Text algorithms to be executed by modifying
run_lynx = True
run_justext = True
run_html2text = True
run_beautifulsoup = True
run_inscriptis = True
In url_list.txt the URLs to be parsed can be specified by adding them to the file, one per line with no additional formatting. URLs need to be complete (including http:// or https://)
e.g.
http://www.informationscience.ch
https://en.wikipedia.org/wiki/Information_science
...
Changelog
see Release notes.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for inscriptis-0.0.4.1.1-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | b3f77c773bb0453c8f775c508ca1868650c6268dcfdd153528401fd2b051ea9b |
|
| MD5 | 524a1747658ad89e5d6f07aa4694b3bd |
|
| BLAKE2b-256 | 26435668de7fe85bb1793285375cf3eeb24e0623b502c42c3f66e529480e8533 |
