Clean Links

Decide whether two links point at the same resource — so you can de-duplicate and count links shared across feeds (a Nuzzel-style use case). Shorteners hide the destination and per-share tracking parameters mean the final URLs rarely match as strings; clean-links follows the redirects, strips the cruft, and compares what's left.

Install with:

pip install clean_links

Are two shortened, tracking-tagged links the same article?

>>> from clean_links import are_equivalent_sync
>>> are_equivalent_sync(
...     "https://bit.ly/some-short-link",
...     "https://trib.al/another-one",
... )
True

Or count how many links in a batch point at the same thing:

>>> from clean_links import group_sync
>>> groups = group_sync([
...     "https://bit.ly/some-short-link",
...     "https://trib.al/another-one",
...     "https://example.com/unrelated",
... ])
>>> {key: len(members) for key, members in groups.items()}
{'https://www.bloomberg.com/news/articles/...': 2, 'https://example.com/unrelated': 1}

These calls make live network requests to follow the redirects. The library is async-first — drop the _sync suffix for the coroutine API — and an Engine with a SqliteStore resolves each link once and reuses it across runs.

See the documentation here.