Fast and robust extraction of original and updated publication dates from web pages.
Project description
- Code:
- Documentation:
- Issue tracker:
Find original and updated publication dates of any web page. From the command-line or within Python, all the steps needed from web page download to HTML parsing, scraping, and text analysis are included.
In a nutshell, with Python:
>>> from htmldate import find_date
>>> find_date('http://blog.python.org/2016/12/python-360-is-now-available.html')
'2016-12-23'
>>> find_date('https://netzpolitik.org/2016/die-cider-connection-abmahnungen-gegen-nutzer-von-creative-commons-bildern/', original_date=True)
'2016-06-23'
On the command-line:
$ htmldate -u http://blog.python.org/2016/12/python-360-is-now-available.html
'2016-12-23'
Features
htmldate finds original and updated publication dates of web pages using heuristics on HTML code and linguistic patterns. URLs, HTML files, or HTML trees are given as input. It provides following ways to date a HTML document:
Markup in header: common patterns are used to identify relevant elements (e.g. link and meta elements) including Open Graph protocol attributes and a large number of CMS idiosyncrasies
HTML code: The whole document is then searched for structural markers: abbr and time elements as well as a series of attributes (e.g. postmetadata)
Bare HTML content: A series of heuristics is run on text and markup:
in fast mode the HTML page is cleaned and precise patterns are targeted
in extensive mode all potential dates are collected and a disambiguation algorithm determines the best one
The output is thouroughly verified in terms of plausibility and adequateness and the library outputs a date string, corresponding to either the last update or the original publishing statement (the default), in the desired format (defaults to ISO 8601 YMD format).
Should be compatible with all recent versions of Python (currently 3.4 to 3.9)
Designed to be computationally efficient and used in production on millions of documents
Batch processing of a list of URLs
Switch between original and updated date
Markup-based extraction is multilingual by nature, text-based refinements for better coverage currently support German, English and Turkish.
Performance
225 web pages containing identifiable dates (as of 2020-07-29) |
|||||
---|---|---|---|---|---|
Python Package |
Precision |
Recall |
Accuracy |
F-Score |
Time |
articleDateExtractor 0.20 |
0.817 |
0.635 |
0.556 |
0.714 |
6.8 |
date_guesser 2.1.4 |
0.809 |
0.553 |
0.489 |
0.657 |
40.0 |
goose3 3.1.6 |
0.887 |
0.441 |
0.418 |
0.589 |
15.5 |
htmldate 0.7.0 (fast) |
0.903 |
0.907 |
0.827 |
0.905 |
2.4 |
htmldate[all] 0.7.0 (extensive) |
0.889 |
1.000 |
0.889 |
0.941 |
3.8 |
newspaper 0.2.8 |
0.888 |
0.407 |
0.387 |
0.558 |
81.6 |
news-please 1.5.3 |
0.823 |
0.660 |
0.578 |
0.732 |
69.6 |
For complete results and explanations see the evaluation page.
Installation
This Python package is tested on Linux, macOS and Windows systems, it is compatible with Python 3.4 upwards. It is available on the package repository PyPI and can notably be installed with pip or pipenv:
$ pip install htmldate # pip3 install on systems where both Python 2 and 3 are installed
$ pip install --upgrade htmldate # to make sure you have the latest version
$ pip install git+https://github.com/adbar/htmldate.git # latest available code (see build status above)
A few additional libraries can be installed to enhance coverage and speed, most importantly ciso8601 and regex (for speed) as well as dateparser (to go beyond the current focus on English or German). They may not work on all platforms and have thus been singled out although installation is recommended:
$ pip install htmldate[all] # install with all additional functionality
You can also install or update the packages separately, htmldate will detect which ones are present on your system and opt for the best available combination.
For faster processing of downloads you may also consider installing the cchardet package as well (currently not working on some macOS versions).
For infos on dependency management of Python packages see this discussion thread
With Python
>>> from htmldate import find_date
>>> find_date('http://blog.python.org/2016/12/python-360-is-now-available.html')
'2016-12-23'
Complete screening of the document with the extensive_search parameter:
>>> find_date('https://creativecommons.org/about/')
'2017-08-11' # has been updated since
>>> find_date('https://creativecommons.org/about/', extensive_search=False)
>>>
Already parsed HTML (that is a LXML tree object):
# simple HTML document as string
>>> htmldoc = '<html><body><span class="entry-date">July 12th, 2016</span></body></html>'
>>> find_date(htmldoc)
'2016-07-12'
# parsed LXML tree
>>> from lxml import html
>>> mytree = html.fromstring('<html><body><span class="entry-date">July 12th, 2016</span></body></html>')
>>> find_date(mytree)
'2016-07-12'
Change the output to a format known to Python’s datetime module, the default being %Y-%m-%d:
>>> find_date('https://www.gnu.org/licenses/gpl-3.0.en.html', outputformat='%d %B %Y')
'18 November 2016' # may have changed since
Although the time delta between original publication and “last modified” info is usually a matter of hours or days, it can be useful to prioritize the original publication date:
>>> find_date('https://netzpolitik.org/2016/die-cider-connection-abmahnungen-gegen-nutzer-von-creative-commons-bildern/', original_date=True) # modified behavior
'2016-06-23'
On the command-line
$ htmldate -u http://blog.python.org/2016/12/python-360-is-now-available.html
'2016-12-23'
$ htmldate --help
htmldate [-h] [-v] [-f] [--original] [-min MINDATE] [-max MAXDATE] [-i INPUTFILE] [-u URL]
- optional arguments:
- -h, --help
show this help message and exit
- -v, --verbose
increase output verbosity
- -f, --fast
fast mode: disable extensive search
- --original
original date prioritized
- -min, --mindate MINDATE
earliest acceptable date (YYYY-MM-DD)
- -max, --maxdate MAXDATE
latest acceptable date (YYYY-MM-DD)
- -i, --inputfile INPUTFILE
name of input file for batch processing (similar to wget -i)
- -u, --URL URL
custom URL download
The batch mode -i takes one URL per line as input and returns one result per line in tab-separated format:
$ htmldate --fast -i list-of-urls.txt
License
htmldate is distributed under the GNU General Public License v3.0. If you wish to redistribute this library but feel bounded by the license conditions please try interacting at arms length, multi-licensing with compatible licenses, or contacting me.
See also GPL and free software licensing: What’s in it for business?
Going further
Online documentation: htmldate.readthedocs.io
If the date is nowhere to be found, it might be worth considering carbon dating the web page, however this is computationally expensive. In addition, datefinder features pattern-based date extraction for texts written in English.
Contributing
Contributions are welcome!
Feel free to file issues on the dedicated page. Thanks to the contributors who submitted features and bugfixes!
Kudos to the following software libraries:
A few patterns are derived from the python-goose, metascraper, newspaper and articleDateExtractor libraries. This module extends their coverage and robustness significantly.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.