Downloads web pages, scrapes main text and comments while preserving some structure, and converts to TXT, CSV, XML & TEI-XML
Project description
- Code:
- Documentation:
- Issue tracker:
Trafilatura downloads web pages, scrapes main text and comments while preserving some structure, and converts to TXT, CSV, XML & TEI-XML. All the operations needed are handled seamlessly.
In a nutshell, with Python:
>>> import trafilatura
>>> downloaded = trafilatura.fetch_url('https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/')
>>> trafilatura.extract(downloaded)
# outputs main content and comments as plain text ...
On the command-line:
$ trafilatura -u "https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/"
# outputs main content and comments as plain text ...
Description
This library performs a robust extraction which focuses on the main content, which is usually the part displayed centrally, without the left or right bars, the header or the footer, but including potential titles and comments. Trafilatura can seamlessly download, parse and convert web documents. It scrapes the main body text while preserving part of the text formatting and page structure, a task also known as web scraping, boilerplate removal, DOM-based content extraction, main content identification, or web page cleaning.
Distinguishing between whole page and essential parts can help to alleviate many quality problems related to web texts as it can help with the noise consisting of recurring elements (headers and footers, ads, links/blogroll, etc.) It has to be precise enough not to miss texts or discard valid documents, it also has to be reasonably fast, as it is expected to run in production on millions of documents.
Features
Seamless download and extraction: URLs, HTML files or parsed HTML trees as input
Focus on main text and/or comments
Formatting and structural elements preserved: paragraphs, titles, lists, quotes, code, line breaks, in-line text formatting (experimental)
Output in plain text (minimal formatting), CSV (with metadata, tab-separated values) or XML format (for metadata and structure)
Extraction of metadata (currently title and date, more to come)
Computationally efficient (relies on lxml)
Robust extraction and generic jusText algorithm used as fallback
Optional language detection on the extracted content
Installation
This Python package is tested on Linux, macOS and Windows systems, it is compatible with Python 3.5 upwards (see install Python guide). It is available on the package repository PyPI and can notably be installed with pip or pipenv:
$ pip install trafilatura # pip3 install on systems where both Python 2 and 3 are installed
$ pip install -U trafilatura # to make sure you have the latest version
$ pip install git+https://github.com/adbar/trafilatura.git # latest available code (see build status above)
A few additional libraries can be installed for extended functionality and faster processing: extraction of publication date (htmldate), language detection (langid), and faster processing of downloads (cchardet, currently not working on some macOS versions).
$ pip install trafilatura[metadata] # metadata extraction
$ pip install trafilatura[all] # all additional functionality
You can also install or update the packages separately, trafilatura will detect which ones are present on your system and opt for the best available combination.
For infos on dependency management of Python packages see this discussion thread
Usage with Python
>>> import trafilatura
>>> downloaded = trafilatura.fetch_url('https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/')
>>> downloaded is None # assuming the download was successful
False
>>> result = trafilatura.extract(downloaded) # trafilatura.process_record is deprecated but works
>>> print(result)
# newlines preserved, TXT output ...
>>> result = trafilatura.extract(downloaded, xml_output=True)
>>> print(result)
# some formatting preserved in basic XML structure ...
The only required argument is the input document (here a downloaded HTML file), the rest is optional.
The inclusion of tables and comments can be deactivated at a function call. The use of a fallback algorithm (currently jusText) can also be bypassed in fast mode:
>>> result = trafilatura.extract(downloaded, include_comments=False) # no comments in output
>>> result = trafilatura.extract(downloaded, include_tables=False) # skip tables examination
>>> result = trafilatura.extract(downloaded, no_fallback=True) # skip justext algorithm used as fallback
This values combined probably provide the fastest execution times:
>>> result = trafilatura.extract(downloaded, include_comments=False, include_tables=False, no_fallback=True)
The input can consist of a previously parsed tree (i.e. a lxml.html object), which is then handled seamlessly:
>>> from lxml import html
>>> mytree = html.fromstring('<html><body><article><p>Here is the main text. It has to be long enough in order to bypass the safety checks. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.</p></article></body></html>')
>>> trafilatura.extract(mytree)
'Here is the main text. It has to be long enough in order to bypass the safety checks. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.\n'
Experimental feature: the target language can also be set using 2-letter codes (ISO 639-1), there will be no output if the detected language of the result does not match and no such filtering if the identification component has not been installed (see above for installation instructions).
>>> result = trafilatura.extract(downloaded, url, target_language='de')
All currently available options, along with their default values:
>>>> trafilatura.extract(downloaded, url=None, record_id='0001', no_fallback=False, include_comments=True, csv_output=False, xml_output=False, tei_output=False, tei_validation=False, target_language=None, include_tables=True, include_formatting=False)
For further configuration see the variables in settings.py and re-compile the package locally.
On the command-line
A command-line interface is included, for general instructions see Comment Prompt (tutorial for Windows systems), How to use the Terminal command line in macOS, or An introduction to the Linux Terminal.
URLs can be used directly (-u/--URL):
$ trafilatura -u https://de.creativecommons.org/index.php/was-ist-cc/
$ # outputs main content in plain text format ...
$ trafilatura --xml --URL "https://github.blog/2019-03-29-leader-spotlight-erin-spiceland/"
$ # outputs main text with basic XML structure ...
You can also pipe a HTML document (and response body) to trafilatura:
$ cat myfile.html | trafilatura # use the contents of an already existing file
$ wget -qO- "https://de.creativecommons.org/index.php/was-ist-cc/" | trafilatura # use a custom download
The -i/--inputfile option allows for bulk download and processing of a list of URLs from a file listing one link per line. Beware that there should be a tacit scraping etiquette and that a server may block you after the download a certain number of pages from the same website/domain in a short period of time. In addition, some website may block the requests user-agent. Thus, trafilatura waits a few seconds per default between requests.
For all usage instructions see trafilatura -h:
usage: trafilatura [-h] [-f] [--formatting] [-i INPUTFILE] [--nocomments] [--notables] [--xml] [--xmltei] [-u URL] [-v]
- optional arguments:
- -h, --help
show this help message and exit
- -f, --fast
fast (without fallback detection)
- --formatting
include text formatting (bold, italic, etc.)
- -i INPUTFILE, --inputfile INPUTFILE
name of input file for batch processing
- --nocomments
don’t output any comments
- --notables
don’t output any table elements
- --csv
CSV output
- --xml
XML output
- --xmltei
XML TEI output
- --validate
validate TEI output
- -u URL, --URL URL
custom URL download
- -v, --verbose
increase output verbosity
License
trafilatura is distributed under the GNU General Public License v3.0
Going further
Online documentation: trafilatura.readthedocs.io
Trafilatura: Italian word for wire drawing.
Tutorial video in German by Simon Meier-Vieracker: Content von Webseiten laden mit Trafilatura.
Tutorials in German by Noah Bubenhofer: Download von Web-Daten & Daten aufbereiten und verwalten.
Roadmap
[-] Duplicate detection at sentence, paragraph and document level using a least recently used (LRU) cache
[-] XML output compatible with the recommendations of the Text Encoding Initiative
[-] Metadata integration
[-] Language detection on the extracted content
[-] Preservation of in-line text formatting (bold, italic, etc.)
[ ] Configuration and extraction parameters
Contributing
Thanks to these contributors who submitted features and bugfixes:
Contributions are welcome!
Feel free to file bug reports on the issues page.
Kudos to the following software libraries:
Alternatives
Most corresponding Python packages are not actively maintained, the following alternatives exist:
dragnet features combined and machine-learning approaches, but requires many dependencies as well as extensive tuning
goose can extract information for embedded content but doesn’t preserve markup and is not maintained
html2text converts HTML pages to Markup language and thus keeps the structure, though it doesn’t focus on main text extraction
newspaper is mostly geared towards newspaper texts, provides additional functions but no structured text or comment extraction.
python-readability cleans the page and preserves some markup but is mostly geared towards news texts
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for trafilatura-0.3.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 69df41ba29dadf571922152c604a816d3265d916d16f9916cdfa04e347c8a325 |
|
MD5 | 5d996624eda8dd205865b9d9bcc488e9 |
|
BLAKE2b-256 | b4122a8e276845a43d74ec124d575673c54fc8e5757de405080f17ca5e6ffacf |