Skip to main content

Fast and robust extraction of original and updated publication dates from URLs and web pages.

Project description

Python package Python versions Documentation Status Code Coverage Downloads JOSS article reference DOI: 10.21105/joss.02439 Code style: black

Logo as PNG image

Find original and updated publication dates of any web page. From the command-line or within Python, all the steps needed from web page download to HTML parsing, scraping, and text analysis are included.

In a nutshell


Demo as GIF image

With Python:

>>> from htmldate import find_date
>>> find_date('http://blog.python.org/2016/12/python-360-is-now-available.html')
'2016-12-23'

On the command-line:

$ htmldate -u http://blog.python.org/2016/12/python-360-is-now-available.html
'2016-12-23'

Features

  • Multilingual, robust and efficient (used in production on millions of documents)

  • URLs, HTML files, or HTML trees are given as input (includes batch processing)

  • Output as string in any date format (defaults to ISO 8601 YMD)

  • Detection of both original and updated dates

  • Compatible with all recent versions of Python

htmldate can examine markup and text. It provides the following ways to date an HTML document:

  1. Markup in header: Common patterns are used to identify relevant elements (e.g. link and meta elements) including Open Graph protocol attributes

  2. HTML code: The whole document is searched for structural markers: abbr or time elements and a series of attributes (e.g. postmetadata)

  3. Bare HTML content: Heuristics are run on text and markup:

  • in fast mode the HTML page is cleaned and precise patterns are targeted

  • in extensive mode all potential dates are collected and a disambiguation algorithm determines the best one

Finally the output is validated and converted to the chosen format.

Performance

1000 web pages containing identifiable dates (as of 2023-11-13 on Python 3.10)

Python Package

Precision

Recall

Accuracy

F-Score

Time

articleDateExtractor 0.20

0.803

0.734

0.622

0.767

5x

date_guesser 2.1.4

0.781

0.600

0.514

0.679

18x

goose3 3.1.17

0.869

0.532

0.493

0.660

15x

htmldate[all] 1.6.0 (fast)

0.883

0.924

0.823

0.903

1x

htmldate[all] 1.6.0 (extensive)

0.870

0.993

0.865

0.928

1.7x

newspaper3k 0.2.8

0.769

0.667

0.556

0.715

15x

news-please 1.5.35

0.801

0.768

0.645

0.784

34x

For complete results and explanations see the evaluation page.

Installation

This Python package is tested on Linux, macOS and Windows systems; it is compatible with Python 3.6 upwards. It is available on the package repository PyPI and can notably be installed with pip (pip3 where applicable): pip install htmldate and optionally pip install htmldate[speed].

Documentation

For more details on installation, Python & CLI usage, please refer to the documentation: htmldate.readthedocs.io

License

htmldate is distributed under the GNU General Public License v3.0. If you wish to redistribute this library but feel bounded by the license conditions please try interacting at arms length, multi-licensing with compatible licenses, or contacting me.

See also GPL and free software licensing: What’s in it for business?

Author

This effort is part of methods to derive information from web documents in order to build text databases for research (chiefly linguistic analysis and natural language processing). Extracting and pre-processing web texts to the exacting standards of scientific research presents a substantial challenge for those who conduct such research. There are web pages for which neither the URL nor the server response provide a reliable way to find out when a document was published or modified. For more information:

JOSS article reference DOI: 10.21105/joss.02439 Zenodo archive DOI: 10.5281/zenodo.3459599
@article{barbaresi-2020-htmldate,
  title = {{htmldate: A Python package to extract publication dates from web pages}},
  author = "Barbaresi, Adrien",
  journal = "Journal of Open Source Software",
  volume = 5,
  number = 51,
  pages = 2439,
  url = {https://doi.org/10.21105/joss.02439},
  publisher = {The Open Journal},
  year = 2020,
}

You can contact me via my contact page or GitHub.

Contributing

Contributions are welcome!

Feel free to file issues on the dedicated page. Thanks to the contributors who submitted features and bugfixes!

Kudos to the following software libraries:

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

htmldate-1.7.0.tar.gz (54.0 kB view hashes)

Uploaded source

Built Distribution

htmldate-1.7.0-py3-none-any.whl (40.9 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page