Web scraping library and command-line tool for text discovery and retrieval. Downloads web pages, scrapes main text and comments while preserving some structure, and converts to TXT, CSV, JSON and XML
Project description
Description
Trafilatura is a Python package and command-line tool which seamlessly downloads, parses, and scrapes web page data: it can extract metadata, main body text and comments while preserving parts of the text formatting and page structure. The output can be converted to different formats.
Features
- Seamless and parallel online/offline processing:
Download and conversion utilities included
URLs, HTML files or parsed HTML trees as input
- Robust and efficient extraction:
Main text and/or comments
Structural elements preserved: paragraphs, titles, lists, quotes, code, line breaks, in-line text formatting
Extraction of metadata (title, author, date, site name, categories and tags)
- Several output formats supported:
Text (minimal formatting or Markdown)
CSV (with metadata, tab-separated values)
JSON (with metadata)
XML (for metadata and structure) and TEI-XML
- Link discovery and URL lists:
Focused crawling and politeness rules
Support for sitemaps and ATOM/RSS feeds
Efficient and polite processing of URL queues
Blacklisting
Optional language detection on extracted content
Evaluation and alternatives
For more detailed results see the benchmark and evaluation script. To reproduce the tests just clone the repository, install all necessary packages and run the evaluation script with the data provided in the tests directory.
500 documents, 1487 text and 1496 boilerplate segments (2021-06-07) |
|||||
---|---|---|---|---|---|
Python Package |
Precision |
Recall |
Accuracy |
F-Score |
Diff. |
justext 2.2.0 (custom) |
0.870 |
0.584 |
0.749 |
0.699 |
6.1x |
newspaper3k 0.2.8 |
0.921 |
0.574 |
0.763 |
0.708 |
12.9x |
boilerpy3 1.0.2 (article mode) |
0.851 |
0.696 |
0.788 |
0.766 |
4.8x |
goose3 3.1.9 |
0.950 |
0.644 |
0.806 |
0.767 |
18.8x |
baseline (text markup) |
0.746 |
0.804 |
0.766 |
0.774 |
1x |
dragnet 2.0.4 |
0.906 |
0.689 |
0.810 |
0.783 |
3.1x |
readability-lxml 0.8.1 |
0.917 |
0.716 |
0.826 |
0.804 |
5.9x |
news-please 1.5.21 |
0.924 |
0.718 |
0.830 |
0.808 |
60x |
trafilatura 0.8.2 (fast) |
0.925 |
0.868 |
0.899 |
0.896 |
3.9x |
trafilatura 0.8.2 |
0.934 |
0.890 |
0.914 |
0.912 |
8.4x |
External evaluations:
Most efficient open-source library in ScrapingHub’s article extraction benchmark as well as in another independant evaluation on the same data.
Evaluation page in the docs.
Usage and documentation
For more information please refer to the documentation:
Usage: On the command-line, With Python, With R
Python Notebook Trafilatura Overview
For video tutorials see this Youtube playlist:
License
Trafilatura is distributed under the GNU General Public License v3.0. If you wish to redistribute this library but feel bounded by the license conditions please try interacting at arms length, multi-licensing with compatible licenses, or contacting me.
See also GPL and free software licensing: What’s in it for business?
Contributing
Contributions are welcome! Please also feel free to file issues on the dedicated page.
Many thanks to the contributors who submitted features and bugfixes!
Context
Software
Trafilatura: Italian word for wire drawing.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for trafilatura-0.9.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 088222fabe5359538be0877222040991dbd482c3e6c28c6521e9152262c88d99 |
|
MD5 | 636b3b74d3e32b63a4611cc66eacfb38 |
|
BLAKE2b-256 | fab0bbf4d9db2223fbe51c6ac50f98777e0af40deb5ae12bc81c7d363da051cd |