URL extraction, generalization, validation, and filtration made easy.

🚀 About URL Genie

A python package based on research involving over 2 million URLs, designed to handle URLs in a flexible manner for data-driven projects.

Version 2 is a rewrite with an RFC 3986 compliant core and small, explicit functions in place of one generalize() carrying a dozen flags. See Migrating from 1.x if you are upgrading.

✨ Features

• RFC 3986 compliant parsing, normalization and validation.
• Social profile recognition for Facebook, LinkedIn, Twitter/X, Instagram and YouTube.
• Email, phone and social extraction from free text, with validation.
• Domain-scoped email filtering, to drop contacts unrelated to a site.
• Duplicate reduction: every spelling of a profile collapses to one canonical URL.
• Public suffix validation against the bundled PSL snapshot -- no network access.
• Pure functions: no hidden state, safe to use across threads and processes.

⚙️ Installation

python -m pip install urlgenie

Depends only on tldextract, used offline against its bundled public suffix list.

♨️ Usage

from urlgenie import generalize, generalize_social, extract_social_handle

print(generalize("cnn.com/sports/about?a=b#c"))
>> https://cnn.com/sports/about

print(generalize("facebook.com.br/@Ahmed.Khatib"))
>> https://www.facebook.com/ahmedkhatib

print(generalize_social("x.com/elonmusk"))
>> https://twitter.com/elonmusk

print(extract_social_handle("x.com/elonmusk").handle)
>> elonmusk

Everything returns None on invalid input rather than a magic string, so use generalize(url) or "Bad Url" if you want a sentinel:

df["clean"] = df["url"].apply(lambda u: generalize(u) or "Bad Url")

For a one-off batch, generalize_many keeps every entry in place -- an invalid URL comes back as None rather than being silently dropped, so nothing shifts out of alignment with your input:

from urlgenie import generalize_many

for original, result in generalize_many("facebook.com,facebook/com,dummy.com,instagram.com/ahmed"):
    print(f"{original:22} -> {result}")

>> facebook.com           -> https://facebook.com
>> facebook/com           -> None
>> dummy.com              -> https://dummy.com
>> instagram.com/ahmed    -> https://www.instagram.com/ahmed

For a one-off run where you only want the URLs that came out clean and do not need to know which input produced which, pass drop_invalid=True:

generalize_many("facebook.com,facebook/com,dummy.com,instagram.com/ahmed", drop_invalid=True)
>> [('facebook.com', 'https://facebook.com'), ('dummy.com', 'https://dummy.com'), ('instagram.com/ahmed', 'https://www.instagram.com/ahmed')]

print(generalize("random.haz/somePath") or "Bad Url")
>> Bad Url

Extracting contacts from a page

from urlgenie import extract_contacts, validate_contacts

result = extract_contacts(scraped_text)

print(result.emails)
>> {'sales@example.com', 'someone@gmail.com'}

print(result.facebook)
>> {'https://www.facebook.com/example'}

print(result.phones)
>> {'+14155552671'}

#-Keep only emails belonging to the site being scraped-#
validated = validate_contacts(result, url="https://www.example.com/contact")

print(validated.emails)
>> {'sales@example.com'}

📋 API

🔀 Migrating from 1.x

Version 2 removes the UrlGenie class. Import the functions directly.

# 1.x
from urlgenie import UrlGenie
genie = UrlGenie(bad_url = "Bad Url", proper_tlds = True)
print(genie.generalize("fb.com/@ahmedkhatib"))
>> https://www.facebook.com/ahmedkhatib

# 2.x
from urlgenie import generalize
print(generalize("fb.com/@ahmedkhatib") or "Bad Url")
>> https://www.facebook.com/ahmedkhatib

Behaviour changes worth knowing:

• URLs 1.x silently rejected now parse: TLDs longer than four characters, subdomain labels longer than ten, ports, IDN hosts, IP literals, and example.com?a=b. Measured against the Tranco top 1M, 1.x rejected 4.72% of real domains; 2.x rejects 0.
• Twitter handles follow Twitter's actual rules (15 characters, no periods), so some strings 1.x accepted are now rejected.
• Facebook handles have periods stripped, since Facebook ignores them when resolving profiles. LinkedIn and Instagram periods are significant and preserved.

🧩 Adding a social pattern

Every pattern is a separate named Rule in urlgenie/config.py. Rules are tried in order and the first yielding a non-reserved handle wins, so supporting a new URL shape is one line and touches nothing else:

FACEBOOK_RULES = (
    Rule("fb_query_id",  re.compile(r"[?&](?:id|gid|__user)=(?P<id>\d{5,})", _I)),
    Rule("fb_media_set", re.compile(r"[?&]set=a\.(?:\d+\.)*(?P<id>\d{5,})", _I)),
    ...
)

A rule exposes an id or handle named group; an optional subdir group or attribute sets the canonical path prefix. transform= handles the rare case needing real logic, such as LinkedIn's legacy /pub/ to /in/ slug conversion.

📐 RFC 3986

urlgenie/uri.py implements scheme and host case normalization (§3.1, §3.2.2), IDNA punycoding, default-port removal (§3.2.3), dot-segment resolution (§5.2.4) and percent-encoding normalization (§2.1, §6.2.2.2). The last is idempotent, unlike quote(unquote(url)), which corrupts %2520 and literal %.

⚡ Performance

Roughly 60,000 URLs per second single-threaded (~17s for one million), measured over a corpus of one million distinct URLs. Every function is pure, so multiprocessing.Pool scales it linearly.

python tools/benchmark.py 1000000

🧪 Tests

python -m pytest tests -q      # 284 tests

tests/sample.csv is the reference sheet and is treated as the spec: every row is asserted for both the generalized URL and the extracted handle.

📖 Resources

• Sample Sheet
• Social Research Doc

⭐ Love It? Star It!

Just a simple click but would help me out ;)