urlfinderlib
This is a Python (3.10+) library for finding URLs in documents and checking their validity.
Supported Documents
Extracts URLs from the following types of documents:
- Binary files (finds URLs within strings)
- CSV files
- HTML files
- iCalendar/vCalendar files
- PDF files
- Text files (ASCII or UTF-8)
- XML files
Every extracted URL is validated such that it contains a domain with a valid TLD (or a valid IP address) and does not contain any invalid characters.
URL Permutations
This was originally written to accommodate finding both valid and obfuscated or slightly malformed URLs used by malicious actors and using them as indicators of compromise (IOCs). As such, the extracted URLs will also include the following permutations:
- URL with any Unicode characters in its domain
- URL with any Unicode characters converted to its IDNA equivalent
For both domain variations, the following permutations are also returned:
- URL with its path %-encoded
- URL with its path %-decoded
- URL with encoded HTML entities in its path
- URL with decoded HTML entities in its path
- URL with its path %-decoded and HTML entities decoded
Child URLs
This library also attempts to extract or decode child URLs found in the paths of URLs. The following formats are supported:
- Barracuda protected URLs
- Base64-encoded URLs found within the URL's path
- Google redirect URLs
- Mandrill/Mailchimp redirect URLs
- Outlook Safe Links URLs
- Proofpoint protected URLs
- URLs found in the URL's path query parameters
Basic usage
from urlfinderlib import find_urls
with open('/path/to/file', 'rb') as f:
print(find_urls(f.read())
base_url Parameter
If you are trying to find URLs inside of an HTML file, the paths in the URLs are often relative to their location on the server hosting the HTML. You can use the base_url parameter in this case to extract these "relative" URLs.
from urlfinderlib import find_urls
with open('/path/to/file', 'rb') as f:
print(find_urls(f.read(), base_url='http://example.com')
3 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file urlfinderlib-0.25.3.tar.gz.
File metadata
- Download URL: urlfinderlib-0.25.3.tar.gz
- Upload date:
- Size: 16.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e149159f64641494e24f0bb0b6f556925152411f6f4cd25137d0bff0b3a1b992
|
|
| MD5 |
611a5eda6e05d3e9e25054fdbde73c6b
|
|
| BLAKE2b-256 |
3d26ddb3fc9bad25068abb53ddecdca9379b4e5329b262f07bb7f806f71e8127
|
Provenance
The following attestation bundles were made for urlfinderlib-0.25.3.tar.gz:
Publisher:
pypi.yml on ACE-Collective/urlfinderlib
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
urlfinderlib-0.25.3.tar.gz -
Subject digest:
e149159f64641494e24f0bb0b6f556925152411f6f4cd25137d0bff0b3a1b992 - Sigstore transparency entry: 2719365562
- Sigstore integration time:
-
Permalink:
ACE-Collective/urlfinderlib@0af7d1b606651506a5d7abd82643cbb38c4914c9 -
Branch / Tag:
refs/heads/main - Owner: https://github.com/ACE-Collective
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
pypi.yml@0af7d1b606651506a5d7abd82643cbb38c4914c9 -
Trigger Event:
push
-
Statement type:
File details
Details for the file urlfinderlib-0.25.3-py3-none-any.whl.
File metadata
- Download URL: urlfinderlib-0.25.3-py3-none-any.whl
- Upload date:
- Size: 22.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3c7b5ca558bd7ef920783da906567c81bcbc4789c21078e291c74894c01bb5dd
|
|
| MD5 |
ebc6b7157de037cf369cfb81c85d244b
|
|
| BLAKE2b-256 |
cc8a484f9ecb392bba40da953d255e32a7bb206122f6b6463b3516e2975add1d
|
Provenance
The following attestation bundles were made for urlfinderlib-0.25.3-py3-none-any.whl:
Publisher:
pypi.yml on ACE-Collective/urlfinderlib
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
urlfinderlib-0.25.3-py3-none-any.whl -
Subject digest:
3c7b5ca558bd7ef920783da906567c81bcbc4789c21078e291c74894c01bb5dd - Sigstore transparency entry: 2719365724
- Sigstore integration time:
-
Permalink:
ACE-Collective/urlfinderlib@0af7d1b606651506a5d7abd82643cbb38c4914c9 -
Branch / Tag:
refs/heads/main - Owner: https://github.com/ACE-Collective
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
pypi.yml@0af7d1b606651506a5d7abd82643cbb38c4914c9 -
Trigger Event:
push
-
Statement type:
