urlstd 2021.10.25.dev1

Python implementation of the WHATWG URL Standard

url-standard

urlstd is a Python implementation of the WHATWG URL Standard.

This library provides URL class, URLSearchParams class, and low-level APIs that comply with the URL specification.

API

• URL class

  • class urlstd.parse.URL(url: str, base: Optional[str] = None)
    • href: readonly property href: str
    • origin: readonly property origin: str
    • protocol: property protocol: str
    • username: property username: str
    • password: property password: str
    • host: property host: str
    • hostname: property hostname: str
    • port: property port: str
    • pathname: property pathname: str
    • search: property search: str
    • searchParams: readonly property search_params: URLSearchParams
    • hash: property hash: str

• URLSearchParams class

  • class urlstd.parse.URLSearchParams(init: Optional[Union[str, Sequence[Sequence[Union[str, int, float]]], Dict[str, Union[str, int, float]], URLRecord, URLSearchParams]] = None)
    • append: append(name: str, value: Union[str, int, float]) -> None
    • delete: delete(name: str) -> None
    • get: get(name: str) -> Optional[str]
    • getAll: get_all(name: str) -> Tuple[str, ...]
    • has: has(name: str) -> bool
    • set: set(name: str, value: Union[str, int, float]) -> None
    • sort: sort() -> None

• Low-level APIs

  • URL parser

    • urlstd.parse.parse_url(urlstring: str, base: str = None, encoding: str = "utf-8") -> URLRecord

  • basic URL parser

    • class urlstd.parse.BasicURLParser
      • classmethod parse(urlstring: str, base: Optional[URLRecord] = None, encoding: str = "utf-8", url: Optional[URLRecord] = None, state_override: Optional[URLParserState] = None) -> URLRecord

  • URL record

    • class urlstd.parse.URLRecord
      • scheme: property scheme: str = ""
      • username: property username: str = ""
      • password: property password: str = ""
      • host: property host: Optional[Union[str, int, Tuple[int, ...]]] = None
      • port: property port: Optional[int] = None
      • path: property path: Union[List[str], str] = []
      • query: property query: Optional[str] = None
      • fragment: property fragment: Optional[str] = None
      • origin: readonly property origin: Optional[Origin]
      • is special: is_special() -> bool
      • is not special: is_not_special() -> bool
      • includes credentials: includes_credentials() -> bool
      • has an opaque path: has_opaque_path() -> bool
      • cannot have a username/password/port: cannot_have_username_password_port() -> bool
      • URL serializer: serialize_url(exclude_fragment: bool = False) -> str
      • host serializer: serialize_host() -> str
      • URL path serializer: serialize_path() -> str

  • IDNA

    • class urlstd.parse.IDNA
      • domain to ASCII: classmethod domain_to_ascii(domain: str, be_strict: bool = False) -> str

  • Host parsing/Host serializing

    • class urlstd.parse.Host
      • host parser: classmethod parse(host: str, is_not_special: bool = False) -> Union[str, int, Tuple[int, ...]]
      • host serializer: classmethod serialize(host: Union[str, int, Sequence[int]]) -> str

  • percent-decode a string

    • urlstd.parse.string_percent_decode(s: str) -> bytes

  • percent-encode after encoding

    • urlstd.parse.string_percent_encode(s: str, safe: str, encoding: str = "utf-8", space_as_plus: bool = False) -> str

  • application/x-www-form-urlencoded parser

    • urlstd.parse.parse_qsl(query: bytes) -> List[Tuple[str, str]]

  • application/x-www-form-urlencoded serializer

    • urlstd.parse.urlencode(query: Sequence[Tuple[str, str]], encoding: str = "utf-8") -> str

• Compatibility with standard library urllib

  • urlstd.parse.urlparse(urlstring: str, base: str = None, encoding: str = "utf-8", allow_fragments: bool = True) -> urllib.parse.ParseResult

    An alternative to urllib.parse.urlparse(). Parses a string representation of a URL using the basic URL parser, and returns urllib.parse.ParseResult.

Basic Usage

>>> from urlstd.parse import URL
>>> url = URL('?ffi&🌈', 'http://example.org')
>>> url
URL(href='http://example.org/?%EF%AC%83&%F0%9F%8C%88', origin='http://example.org', protocol='http:', username='', password='', host='example.org', hostname='example.org', port='', pathname='/', search='?%EF%AC%83&%F0%9F%8C%88', hash='')
>>> url.search
'?%EF%AC%83&%F0%9F%8C%88'
>>> params = url.search_params
>>> params
URLSearchParams([('ffi', ''), ('🌈', '')])
>>> params.sort()
>>> params
URLSearchParams([('🌈', ''), ('ffi', '')])
>>> url.search
'?%F0%9F%8C%88=&%EF%AC%83='
>>> str(url)
'http://example.org/?%F0%9F%8C%88=&%EF%AC%83='

An alternative to urllib.parse.urlparse():

>>> import html
>>> from urllib.parse import unquote
>>> from urlstd.parse import urlparse
>>> urlparse('?aÿb', 'http://example.org/foo/', encoding='utf-8')
ParseResult(scheme='http', netloc='example.org', path='/foo/', params='', query='a%C3%BFb', fragment='')
>>> unquote('a%C3%BFb')
'aÿb'
>>> urlparse('?aÿb', 'http://example.org/foo/', encoding='windows-1251')
ParseResult(scheme='http', netloc='example.org', path='/foo/', params='', query='a%26%23255%3Bb', fragment='')
>>> unquote('a%26%23255%3Bb', encoding='windows-1251')
'aÿb'
>>> html.unescape('aÿb')
'aÿb'
>>> urlparse('?aÿb', 'http://example.org/foo/', encoding='windows-1252')
ParseResult(scheme='http', netloc='example.org', path='/foo/', params='', query='a%FFb', fragment='')
>>> unquote('a%FFb', encoding='windows-1252')
'aÿb'

Logging

urlstd uses standard library logging for validation error. Change the logger log level of urlstd if needed:

logging.getLogger("urlstd").setLevel(logging.ERROR)

Dependencies

• icupy >= 0.11.0
  • ICU - International Components for Unicode (ICU4C)

Installation

1. Configuring environment variables for icupy (ICU):

   • Windows:

     • Set the ICU_ROOT environment variable to the root of the ICU installation (default is C:\icu). For example, if the ICU is located in C:\icu4c:

       set ICU_ROOT=C:\icu4c
       

       or in PowerShell:

       $env:ICU_ROOT = "C:\icu4c"
       

   • Linux/POSIX:

     • If the ICU is located in a non-regular place, set the PKG_CONFIG_PATH and LD_LIBRARY_PATH environment variables. For example, if the ICU is located in /usr/local:

       export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH
       export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
       

2. Installing from PyPI:

   pip install urlstd
   

Running Tests

To run tests and generate a report:

git clone https://github.com/miute/url-standard.git
cd url-standard
tox -e wpt

See result: tests/wpt/report.html

License

MIT License.