Fast and robust extraction of original and updated publication dates from URLs and web pages.
Project description
- Code:
- Documentation:
- Issue tracker:
Find original and updated publication dates of any web page. From the command-line or within Python, all the steps needed from web page download to HTML parsing, scraping, and text analysis are included.
In a nutshell
With Python:
>>> from htmldate import find_date
>>> find_date('http://blog.python.org/2016/12/python-360-is-now-available.html')
'2016-12-23'
>>> find_date('https://netzpolitik.org/2016/die-cider-connection-abmahnungen-gegen-nutzer-von-creative-commons-bildern/', original_date=True)
'2016-06-23'
On the command-line:
$ htmldate -u http://blog.python.org/2016/12/python-360-is-now-available.html
'2016-12-23'
Features
Compatible with all recent versions of Python (see above)
Multilingual, robust and efficient (used in production on millions of documents)
URLs, HTML files, or HTML trees are given as input (includes batch processing)
Output as string in any date format (defaults to ISO 8601 YMD)
Detection of both original and updated dates
htmldate finds original and updated publication dates of web pages using heuristics on HTML code and linguistic patterns. It provides following ways to date a HTML document:
Markup in header: Common patterns are used to identify relevant elements (e.g. link and meta elements) including Open Graph protocol attributes and a large number of CMS idiosyncrasies
HTML code: The whole document is then searched for structural markers: abbr and time elements as well as a series of attributes (e.g. postmetadata)
Bare HTML content: A series of heuristics is run on text and markup:
in fast mode the HTML page is cleaned and precise patterns are targeted
in extensive mode all potential dates are collected and a disambiguation algorithm determines the best one
Performance
500 web pages containing identifiable dates (as of 2022-02-18) |
|||||
---|---|---|---|---|---|
Python Package |
Precision |
Recall |
Accuracy |
F-Score |
Time |
articleDateExtractor 0.20 |
0.769 |
0.691 |
0.572 |
0.728 |
3.0x |
date_guesser 2.1.4 |
0.738 |
0.544 |
0.456 |
0.626 |
11x |
goose3 3.1.11 |
0.821 |
0.453 |
0.412 |
0.584 |
10x |
htmldate[all] 1.1.0 (fast) |
0.840 |
0.911 |
0.776 |
0.874 |
1x |
htmldate[all] 1.1.0 (extensive) |
0.827 |
0.990 |
0.820 |
0.901 |
1.6x |
newspaper3k 0.2.8 |
0.729 |
0.630 |
0.510 |
0.675 |
8.4x |
news-please 1.5.21 |
0.769 |
0.691 |
0.572 |
0.728 |
19x |
For complete results and explanations see the evaluation page.
Installation
This Python package is tested on Linux, macOS and Windows systems, it is compatible with Python 3.6 upwards. It is available on the package repository PyPI and can notably be installed with pip (pip3 where applicable): pip install htmldate and optionally pip install htmldate[speed].
Documentation
For more details on installation, Python & CLI usage, please refer to the documentation: htmldate.readthedocs.io
License
htmldate is distributed under the GNU General Public License v3.0. If you wish to redistribute this library but feel bounded by the license conditions please try interacting at arms length, multi-licensing with compatible licenses, or contacting me.
See also GPL and free software licensing: What’s in it for business?
Contributing
Contributions are welcome!
Feel free to file issues on the dedicated page. Thanks to the contributors who submitted features and bugfixes!
Kudos to the following software libraries:
A few patterns are derived from the python-goose, metascraper, newspaper and articleDateExtractor libraries. This module extends their coverage and robustness significantly.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.