This package provides a set of functions to measure the similarity between web pages.
Install
The quick way:
pip install html-similarity
How it works?
Structural Similarity
Uses sequence comparison of the html tags to compute the similarity, by default.
We not implement the similarity based on tree edit distance because it is slower than sequence comparison.
structural_similarity accepts an algorithm keyword to pick the comparison strategy:
indel (default): flat tag-sequence comparison using rapidfuzz’s bit-parallel Indel/LCS implementation. Fastest option, but blind to nesting (e.g. moving an element to a different parent without changing the overall tag order won’t affect the score).
pq_gram: tree-structure aware. Compares pq-gram profiles, which approximate Tree Edit Distance in roughly linear time while still capturing parent/child relationships. Slower than indel but catches structural changes that a flat sequence misses.
difflib: legacy flat tag-sequence comparison (the original implementation), kept mainly for benchmarking against indel.
See notebooks/structural_similarity_benchmark.ipynb for a notebook that compares the speed and the structural sensitivity of all three.
Style Similarity
Extracts css classes of each html document and calculates the jaccard similarity of the sets of classes.
Joint Similarity (Structural Similarity and Style Similarity)
The joint similarity metric is calculated as:
k * structural_similarity(document_1, document_2) + (1 - k) * style_similarity(document_1, document_2)
All the similarity metrics takes values between 0 and 1.
Recommendations for joint similarity
Using k=0.3 give use better results. The style similarity gives more information about the similarity rather than the structural similarity.
Examples
Here is a example:
In [1]: html_1 = '''
<h1 class="title">First Document</h1>
<ul class="menu">
<li class="active">Documents</li>
<li>Extra</li>
</ul>
'''
In [2]: html_2 = '''
<h1 class="title">Second document Document</h1>
<ul class="menu">
<li class="active">Extra Documents</li>
</ul>
'''
In [3] from html_similarity import style_similarity, structural_similarity, similarity
In [4]: style_similarity(html_1, html_2)
Out[4]: 1.0
In [7]: structural_similarity(html_1, html_2)
Out[7]: 0.9090909090909091
In [8]: similarity(html_1, html_2)
Out[8]: 0.9545454545454546
References
The idea of sequence comparision was taken from Page Compare.
The other ideas were taken from T. Gowda and C. A. Mattmann, Clustering Web Pages Based on Structure and Style Similarity, 2016 IEEE 17th International Conference on Information Reuse and Integration (IRI), Pittsburgh, PA, 2016, pp. 175-180.
Use case Clustering web pages based on structure and style similarity
Thanks
1 maintainer
Release files for html-similarity 0.5.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| html_similarity-0.5.0.tar.gz | 11.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| html_similarity-0.5.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 22.1 kB
Release files / html_similarity-0.5.0.tar.gz
| Download URL | html_similarity-0.5.0.tar.gz |
|---|---|
| Size | 11.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ccbee139dd27d213546af10e608438b3c698258e000fa8f3ea9d2d9dbaf87e8d
|
|
BLAKE2b-256 checksum How to use checksums |
b1f7ff70237f0707da0f50bced78566f36f2ab6eee645a1723517c519fa433dd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.0
|
Release files / html_similarity-0.5.0-py3-none-any.whl
| Download URL | html_similarity-0.5.0-py3-none-any.whl |
|---|---|
| Size | 10.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1dd4f3156558038416946e66d04342036561dea1dd69b8e818e9cda0157c5254
|
|
BLAKE2b-256 checksum How to use checksums |
6495b044bc81ad7a3281b1803c71688288b38044abbea432db257161381f487d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.0
|
