pywhatwgurl - Python WHATWG URL Parser

Pure Python implementation of the WHATWG URL Standard. The goal is a small, spec-faithful library for parsing, serializing, and manipulating URLs.

pywhatwgurl is listed as a complete Python implementation in the official WHATWG URL repository.

Status

• 100% WHATWG URL Standard conformance for core URL parsing
• Full URL and URLSearchParams API implementation
• See Conformance for IDNA limitations

Installation

Requires Python 3.10+.

pip install pywhatwgurl

Quickstart

from pywhatwgurl import URL

url = URL("https://user:pass@example.com:8080/path?q=1#frag")

url.hostname   # 'example.com'
url.port       # '8080'
url.pathname   # '/path'
url.search     # '?q=1'
url.hash       # '#frag'
str(url)       # 'https://user:pass@example.com:8080/path?q=1#frag'

URLSearchParams works just like the browser API:

from pywhatwgurl import URLSearchParams

params = URLSearchParams("a=1&b=2&a=3")
params.get("a")        # '1'
params.get_all("a")    # ('1', '3')
params.set("b", "42")
str(params)            # 'a=1&b=42&a=3'

For full API details, see the documentation.

Development

To set up a local development environment, use uv (recommended) or pip:

# Clone and install dev dependencies
uv sync --dev

# Run the test suite
uv run pytest

If you prefer pip:

python -m venv .venv && source .venv/bin/activate
pip install -e .
pip install pytest ruff mypy pre-commit interrogate pip-audit cyclonedx-bom
pytest

Conformance

This library targets 100% conformance with the WHATWG URL Standard for core URL parsing, validated against the official Web Platform Tests.

IDNA Limitations

IDNA tests are marked as expected failures (xfail) because the Python idna library follows stricter RFC 5891/5892 rules than the WHATWG URL Standard's lenient UTS46 processing.

Why not fix these?

1. No Python WHATWG-compliant IDNA implementation exists
2. Even non-Python implementations with custom IDNA handling still fall short of full compliance
3. Real-world domains work correctly — failures are obscure edge cases

For details, see tests/conformance/README.md.

Supply Chain Security

• SBOM: A Software Bill of Materials (CycloneDX JSON) is automatically generated for every release and attached to the GitHub Release.
• Audit: All runtime dependencies are scanned for known vulnerabilities using pip-audit in our CI pipeline.

Roadmap

• ✅ Implement URL parsing/serialization per WHATWG URL Standard
• ✅ Validate against the official URL test suite (100% conformance)
• Ship a minimal, typed API suitable for frameworks and tooling

WPT URL test data

To pull down the pinned WPT URL resources, use util/wpt_url_test_data.py:

python util/wpt_url_test_data.py download \
  --dest_dir tests/conformance/data \
  --commit <WPT_URL_COMMIT>

The script downloads the WPT URL JSON resources (including urltestdata, setters, percent-encoding, toascii, and IDNA fixtures), preserves comments, validates schemas, and writes a metadata file with the pinned commit. The scheduled workflow .github/workflows/fetch_test_data.yml runs the same script via the composite action and is keyed on the pinned commit; adjust WPT_URL_COMMIT/WPT_TEST_DATA_PATH in the workflow env if you need a different pin.

Updating the pinned WPT commit

• A helper workflow, .github/workflows/update_wpt_url.yml, can be triggered manually (or waits for its weekly schedule) to fetch the latest url/ commit from WPT, bump all pins, refresh the fixtures, and open a PR.
• To bump manually without the bot:
  1. Get the latest url/ commit: NEW_COMMIT=$(curl -s https://api.github.com/repos/web-platform-tests/wpt/commits?path=url&per_page=1 | jq -r '.[0].sha')
  2. Set WPT_URL_COMMIT to that value in .github/workflows/main.yml and .github/workflows/fetch_test_data.yml, and update the defaults in .github/workflows/actions/fetch_wpt_url_test_data/action.yml and util/wpt_url_test_data.py to match.
  3. Refresh fixtures: python util/wpt_url_test_data.py download --dest_dir tests/conformance/data --commit "$NEW_COMMIT"
  4. Commit the workflow and data changes together.

Building

To build the package distribution (wheel and sdist):

uv build

The artifacts will be generated in the dist/ directory.

Versioning is dynamic and derived from Git tags (e.g., 0.1.0 or 0.1.dev1+...).

Documentation

To build and serve the documentation locally:

# Install docs dependencies
uv sync --group docs

# Serve locally (with live reload)
uv run mkdocs serve

# Build static site
uv run mkdocs build

Documentation is automatically deployed to GitHub Pages when changes are pushed to master.