urlps

Lightweight, secure URL parsing and building library with RFC 3986 compliance. Features comprehensive security protections including SSRF prevention, DNS rebinding detection, path traversal protection, and homograph attack detection.

Installation

pip install urlps

Development setup:

python -m venv .venv
. .venv/Scripts/activate  # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

Search cleanup: repository searches can use .rgignore to skip local/IDE/build artifacts.

Quick Start

from urlps import parse_url, build

# Secure by default - blocks SSRF, private IPs, localhost
url = parse_url("https://api.example.com/data?token=abc#section")
print(url.host)  # api.example.com
print(url.query_params)  # [("token", "abc")]

# Build URLs
url_str = build("https", "example.com", port=8443, path="/api", query="x=1")
# https://example.com:8443/api?x=1

# Immutable with functional updates
url = parse_url("https://example.com/path")
new_url = url.with_host("other.com").with_port(8080)
print(new_url)  # https://other.com:8080/path

# Policy-based validation
strict_url = parse_url("https://example.com", policy="strict")
balanced_url = parse_url("HTTP://EXAMPLE.com", policy="balanced")

Security

parse_url() blocks by default:

• Private IPs (192.168.x.x, 10.x.x.x, 172.16.x.x)
• Localhost and loopback addresses
• Link-local addresses (169.254.x.x)
• .local and .internal domains
• Path traversal patterns (../)
• Double-encoded characters
• Mixed Unicode scripts (homograph attacks)
• URL parser confusion attacks

parse_url(..., policy="strict") additionally blocks:

• Query parameter injection
• Dangerous ports (commonly exploited)
• Non-canonical URL forms (filter bypass prevention)
• Credentials in URL userinfo by default (user:pass@host)

Use parse_url_unsafe() for internal/development URLs:

from urlps import SecurityPolicy, parse_url_unsafe

dev_url = parse_url_unsafe("http://localhost:3000/api")
internal = parse_url_unsafe("http://192.168.1.100/metrics")

# If policy is passed, parse_url_unsafe uses it exactly.
trusted_policy = SecurityPolicy.internal(check_dns=True)
internal_checked = parse_url_unsafe("http://intranet.local/service", policy=trusted_policy)

Need selective hardening? Use policy presets:

• policy="strict": maximum protections, DNS connect checks fail-closed by default
• policy="balanced" (default): fewer false positives, DNS connect checks fail-open by default
• policy="internal": trusted/internal traffic

DNS connect behavior can be customized per policy:

from urlps import SecurityPolicy, parse_url

policy = SecurityPolicy.strict(check_dns=True, dns_fail_open_on_connect_error=True)
url = parse_url("https://api.example.com", policy=policy)

Recommended for multi-tenant or concurrent applications: inject a dedicated DNS limiter.

from urlps import parse_url
from urlps._security.dns_guard import DNSRateLimiter, DNSRateLimiterConfig

limiter = DNSRateLimiter(
    DNSRateLimiterConfig(max_lookups_per_second=20, max_lookups_per_host=50)
)

url = parse_url(
    "https://api.example.com",
    policy="strict",
    check_dns=True,
    dns_rate_limiter=limiter,
)

Core Features

Immutable URL Objects

from urlps import parse_url

url = parse_url("https://user:pass@example.com:8080/path?token=abc")
print(url.netloc)         # user:pass@example.com:8080
print(url.effective_port) # 8080

# with_* methods return new URL objects
url2 = url.with_netloc("admin@example.com")
url3 = url.with_host("other.com").with_port(443).with_path("/api")
url4 = url.with_query_param("new", "value")
url5 = url.without_query_param("token")

Security Checks

from urlps import parse_url, InvalidURLError

# SSRF protection (enabled by default)
try:
    parse_url("http://localhost/admin")  # Blocked
except InvalidURLError as e:
    print(f"Rejected: {e}")

# DNS rebinding detection (optional - rate-limited to prevent DoS)
url_dns = parse_url("https://api.example.com/", check_dns=True)

# URL canonicalization
url_raw = parse_url("HTTP://EXAMPLE.COM:80/path?z=1&a=2")
canonical = url_raw.canonicalize()
print(canonical.scheme)  # "http"
print(canonical.host)    # "example.com"
print(canonical.port)    # None (default port removed)
print(canonical.query)   # "a=2&z=1" (sorted)

# Password masking
url = parse_url("https://admin:secret123@api.example.com/")
print(url.as_string(mask_password=True))  # https://admin:***@api.example.com/

Audit Logging

from urlps import set_audit_callback
import logging

def audit_url_parsing(raw_url, parsed_url, exception):
    if exception:
        logging.warning(f"Failed to parse URL: {exception}")
    else:
        logging.info(f"Parsed URL to host: {parsed_url.host}")

set_audit_callback(audit_url_parsing)

Structured event callback:

from urlps import set_audit_event_callback

def on_event(event):
    # event includes: timestamp, level, operation, host, error_code, correlation_id
    print(event)

set_audit_event_callback(on_event)

Component Length Limits

Conservative limits to prevent DoS attacks:

Environment Variables

Override length limits via environment variables:

# PowerShell
$env:URLPS_MAX_URL_LENGTH = "65536"
python -c "import urlps.constants as c; print(c.MAX_URL_LENGTH)"

# Bash
export URLPS_MAX_URL_LENGTH=65536
python -c 'import urlps.constants as c; print(c.MAX_URL_LENGTH)'

Supported variables:

• URLPS_MAX_URL_LENGTH
• URLPS_MAX_SCHEME_LENGTH
• URLPS_MAX_HOST_LENGTH
• URLPS_MAX_PATH_LENGTH
• URLPS_MAX_QUERY_LENGTH
• URLPS_MAX_FRAGMENT_LENGTH
• URLPS_MAX_USERINFO_LENGTH
• URLPS_MAX_IPV6_STRING_LENGTH

API Reference

Main Functions

Note: get_dns_rate_limiter() and reset_dns_rate_limiter() remain available for compatibility, but explicit dns_rate_limiter= injection is preferred.

URL Methods

Cache Management

from urlps import get_cache_info, clear_all_caches

# Get cache statistics
stats = get_cache_info()
print(stats['parser']['normalize_path']['hits'])

# Clear all caches (useful for long-running apps)
previous = clear_all_caches()

Comparison with urllib.parse

Use urllib.parse when: You need zero dependencies and basic parsing is sufficient.

Use urlps when: Security matters, you need RFC 3986 strict compliance, or you want immutable URL objects with ergonomic manipulation methods.

Exceptions

from urlps import InvalidURLError, URLParseError, parse_url

user_input = "https://example.com"

try:
    url = parse_url(user_input)
except URLParseError:
    print("Malformed URL")
except InvalidURLError:
    print("Rejected by security policy")

Exception hierarchy:

• InvalidURLError — Base exception for all URL errors
• URLParseError — Parsing errors
• URLBuildError — Building errors
• HostValidationError / PortValidationError — Component validation errors
• QueryParsingError, FragmentEncodingError, UserInfoParsingError, UnsupportedSchemeError — Specific errors

Running Tests

pytest
pytest -v -k "test_parse"     # Run specific tests
pytest -m ipv6                # Run IPv6 tests
pytest -m idna                # Run IDNA tests

Changelog

See CHANGELOG.md for a summary of every release, and changelogs/ for detailed per-release notes.

License

MIT