upa-url 1.1.0

WHATWG URL Standard compliant URL parser library

upa_url package

This package provides Python bindings for Upa URL – a library compliant with the WHATWG URL standard. This is the same standard followed by modern browsers and JavaScript runtimes such as Bun, Deno, and Node.js.

This package is designed to be as close to the URL standard as possible. It uses the same class names (URL, URLSearchParams), their function names, the same function parameters, and the same behavior.

Installation

pip install upa_url

If the binary wheel is not available for your platform, then you will need a C++ compiler that supports C++17 and CMake to build the Python package.

Getting started

First, you need to import classes:

from upa_url import PSL, URL, URLSearchParams

URL class

The URL class provides a structured way to parse, manipulate, and serialize URLs.

An URL can be parsed using one of two methods:

1. Use the URL constructor. It throws an exception on error:

   try:
       url = URL('https://upa-url.github.io/docs/')
       print(url.href)
   except Exception:
       print('URL parse error')
   

2. Use the URL.parse fucntion. It returns None on error:

   url = URL.parse('docs', 'https://upa-url.github.io')
   if url is not None:
       print(url.href)
   

The components of the parsed URL object can be accessed using getters and setters: href, origin (only get value), protocol, username, password, host, hostname, port, pathname, search and hash. You can also get and change the search parameters using the searchParams getter, which returns the URLSearchParams object associated with the URL:

url = URL.parse('https://example.org')
if url is not None:
    url.searchParams.append('lang', 'lt')
    print(url.href) # https://example.org/?lang=lt

To serialize a parsed URL, use either url.href or str(url).

If you only need to check URL validity, then the URL.canParse function can be used:

if URL.canParse('docs', 'https://upa-url.github.io'):
    print('URL is valid')

URLSearchParams class

The URLSearchParams class provides a structured way to parse, manipulate, and serialize the query string of a URL.

An URLSearchParams object can be created by using a constructor:

1. To create empty: params = URLSearchParams()
2. Create from a string: params = URLSearchParams('lang=lt&id=123')
3. Create from a dictionary or a list:

   params1 = URLSearchParams({'lang': 'lt', 'id': '123'})
   params2 = URLSearchParams([('lang', 'lt'), ['id', '123']])
   

Use get or getAll to retrieve parameter values:

params = URLSearchParams('a=b&a=c&b=10')
print(params.get('a'))    # b
print(params.getAll('a')) # ['b', 'c']

To check for name and optionally value in parameters, use the has function:

print(params.has('a'))      # True
print(params.has('a', 'c')) # True
print(params.has('c'))      # False

Iterate over all parameters:

params = URLSearchParams('a=1&b=2')
# Get all name-value pairs:
for name, value in params:
    print(name, '=', value)
# Get all parameter names
for name in params.keys():
    print(name)
# Get all parameter values
for value in params.values():
    print(value)

Count parameters:

print(params.size) # 2
print(len(params)) # 2

To serialize a URLSearchParams object, use str(params).

There are functions to manipulate search parameters:

1. Add or replace parameters:

   params = URLSearchParams('a=a')
   params.append('a', 'aa')
   params.append('b', 'bb')
   print(params) # a=a&a=aa&b=bb
   params.set('a', '1')
   print(params) # a=1&b=bb
   

2. Remove parameters:

   params = URLSearchParams('a=a&a=aa&b=b&b=bb')
   params.delete('a')
   print(params) # b=b&b=bb
   params.delete('b', 'bb')
   print(params) # b=b
   

3. Sort parameters by name:

   params = URLSearchParams('c=1&b=2&a=3')
   params.sort()
   print(params) # a=3&b=2&c=1
   

PSL class

The PSL class allows getting the public suffix and registrable domain of a given host.

First, you need to create a PSL object and load the Public Suffix List. This list can be downloaded from https://publicsuffix.org/list/public_suffix_list.dat. The downloaded file can be loaded using one of the following methods:

1. Use the load function:

   psl = PSL.load('public_suffix_list.dat')
   if (psl is not None):
       print(psl.public_suffix('upa-url.github.io')) # github.io
   

2. Use the PSL constructor:

   try:
       psl = PSL('public_suffix_list.dat')
       # Use psl
   except Exception:
       print('PSL loading error')
   

The Public Suffix List can be loaded from memory using the push interface:

1. Line by line:

   psl = PSL()
   with open('public_suffix_list.dat', 'r', encoding='utf-8') as f:
       for line in f:
           psl.push_line(line.rstrip())
   if psl.finalize():
       # Use psl
   

2. Using the memory buffer, for example, to load a list from the web:

   import urllib.request
   url = 'https://upa-url.github.io/demo/public_suffix_list.dat'
   psl = PSL()
   with urllib.request.urlopen(url) as response:
       while (chunk := response.read(4096)):
           psl.push(chunk)
   if psl.finalize():
       # Use psl
   

The following examples show how to get a public suffix and a registrable domain:

# Get from the host string
print(psl.public_suffix('abc.ålgård.no')) # xn--lgrd-poac.no
print(psl.registrable_domain('abc.ålgård.no')) # abc.xn--lgrd-poac.no

# Get from the host string and do not convert the output to ASCII
print(psl.public_suffix('abc.ålgård.no', ascii=False)) # ålgård.no
print(psl.registrable_domain('abc.ålgård.no', ascii=False)) # abc.ålgård.no

# Get from the URL
url = URL('https://upa-url.github.io/docs/')
print(psl.public_suffix(url)) # github.io
print(psl.registrable_domain(url)) # upa-url.github.io

License

This package is licensed under the BSD 2-Clause License (see LICENSE file).