upa_url package

This package provides Python bindings for the Upa URL library, which is compliant with the WHATWG URL and URL Pattern standards. These are the same standards followed by modern browsers and JavaScript runtimes such as Bun, Deno, and Node.js.

This package is designed to be as close to the URL and URL Pattern standards as possible. It uses the same class names (URL, URLSearchParams, URLPattern), their function names, the same function parameters, and the same behavior.

Installation

pip install upa_url

If the binary wheel is not available for your platform, then you will need a C++ compiler that supports C++17 and CMake to build the Python package.

Getting started

First, you need to import classes:

from upa_url import PSL, URL, URLPattern, URLSearchParams

URL class

The URL class provides a structured way to parse, manipulate, and serialize URLs.

An URL can be parsed using one of two methods:

1. Use the URL constructor. It throws an exception on error:

   try:
       url = URL('https://upa-url.github.io/docs/')
       print(url.href)
   except Exception:
       print('URL parse error')
   

2. Use the URL.parse fucntion. It returns None on error:

   url = URL.parse('docs', 'https://upa-url.github.io')
   if url is not None:
       print(url.href)
   

The components of the parsed URL object can be accessed using getters and setters: href, origin (only get value), protocol, username, password, host, hostname, port, pathname, search and hash. You can also get and change the search parameters using the searchParams getter, which returns the URLSearchParams object associated with the URL:

url = URL.parse('https://example.org')
if url is not None:
    url.searchParams.append('lang', 'lt')
    print(url.href) # https://example.org/?lang=lt

To serialize a parsed URL, use either url.href or str(url).

If you only need to check URL validity, then the URL.canParse function can be used:

if URL.canParse('docs', 'https://upa-url.github.io'):
    print('URL is valid')

URLSearchParams class

The URLSearchParams class provides a structured way to parse, manipulate, and serialize the query string of a URL.

An URLSearchParams object can be created by using a constructor:

1. To create empty: params = URLSearchParams()
2. Create from a string: params = URLSearchParams('lang=lt&id=123')
3. Create from a dictionary or a list:

   params1 = URLSearchParams({'lang': 'lt', 'id': '123'})
   params2 = URLSearchParams([('lang', 'lt'), ['id', '123']])
   

Use get or getAll to retrieve parameter values:

params = URLSearchParams('a=b&a=c&b=10')
print(params.get('a'))    # b
print(params.getAll('a')) # ['b', 'c']

To check for name and optionally value in parameters, use the has function:

print(params.has('a'))      # True
print(params.has('a', 'c')) # True
print(params.has('c'))      # False

Iterate over all parameters:

params = URLSearchParams('a=1&b=2')
# Get all name-value pairs:
for name, value in params:
    print(name, '=', value)
# Get all parameter names
for name in params.keys():
    print(name)
# Get all parameter values
for value in params.values():
    print(value)

Count parameters:

print(params.size) # 2
print(len(params)) # 2

To serialize a URLSearchParams object, use str(params).

There are functions to manipulate search parameters:

1. Add or replace parameters:

   params = URLSearchParams('a=a')
   params.append('a', 'aa')
   params.append('b', 'bb')
   print(params) # a=a&a=aa&b=bb
   params.set('a', '1')
   print(params) # a=1&b=bb
   

2. Remove parameters:

   params = URLSearchParams('a=a&a=aa&b=b&b=bb')
   params.delete('a')
   print(params) # b=b&b=bb
   params.delete('b', 'bb')
   print(params) # b=b
   

3. Sort parameters by name:

   params = URLSearchParams('c=1&b=2&a=3')
   params.sort()
   print(params) # a=3&b=2&c=1
   

URLPattern class

The object of the URLPattern class contains a URL pattern matcher that can be used to match against URLs or a dictionary of URL components.

An URLPattern object can be created by using a constructor:

1. The constructor without arguments (urlp = URLPattern()) creates an object that matches any URL.
2. Create from a URL string containing pattern syntax for one or more components:

   urlp = URLPattern('http{s}?://:label.lt')
   

3. Create from a relative URL pattern string and base URL:

   urlp = URLPattern('/:id([0-9]+)', 'https://example.org/')
   

4. Create from URL components:

   urlp = URLPattern({'protocol': 'http{s}?', 'hostname': ':label.lt'})
   # With base URL:
   urlp = URLPattern({'hostname': ':label.lt', 'baseURL': 'wss://example.com/'})
   

5. Create URLPattern object for case-insensitive matching:

   urlp = URLPattern('http{s}?://:label.lt/path', {'ignoreCase': True})
   

The constructor parses the pattern string and converts it into a canonical form for each URL component. Each component's canonicalized pattern string can be examined. For example:

urlp = URLPattern('http{s}?://:label.lt:([0-9]+)?/:id([a-z]+)')
print(urlp.protocol) # http{s}?
print(urlp.username) # *
print(urlp.password) # *
print(urlp.hostname) # :label.lt
print(urlp.port) # ([0-9]+)?
print(urlp.pathname) # /:id([a-z]+)
print(urlp.search) # *
print(urlp.hash) # *
print(urlp.hasRegExpGroups) # True

Use the test() method when you need to check if a URL or a dictionary of URL components matches a URL pattern:

urlp = URLPattern('http{s}?://:label.lt:([0-9]+)?/:id([a-z]+)')
print(urlp.test('https://lrt.lt/mediateka')) # True
# With base URL:
print(urlp.test('/123', 'https://lrt.lt/')) # False
print(urlp.test({'pathname': '/programa', 'baseURL': 'https://lrt.lt/'})) # True

The exec() method is similar to the test() method. It accepts the same parameters, but returns more information about the match. If there is no match, exec() returns None. In the case of a match, the exec() returns a dictionary with the following keys:

• inputs. This key's value is an array containing the inputs passed to the exec().
• protocol, username, password, hostname, port, pathname, search, and hash. Its values are dictionaries that correspond to each URL component. Each dictionary has the following keys:
  • input. The value of this key is the part of the input that corresponds to the URL component.
  • groups. The value of this key is a dictionary with keys for each match group in the URL component (if any), and the corresponding matched values in the inputs. Group keys are numbered from 0 for unnamed match groups (such as the wildcard). For named match groups, the key name is the group name.

Example:

urlp = URLPattern('http{s}?://:label.lt:([0-9]+)?/:id([a-z]+)')
res = urlp.exec('http://lrt.lt:8080/mediateka')
print(res['inputs']) # ['http://lrt.lt:8080/mediateka']
print(res['protocol']) # {'input': 'http', 'groups': {}}
print(res['hostname']) # {'input': 'lrt.lt', 'groups': {'label': 'lrt'}}
print(res['port']) # {'input': '8080', 'groups': {'0': '8080'}}
print(res['pathname']) # {'input': '/mediateka', 'groups': {'id': 'mediateka'}}
print(res['search']) # {'input': '', 'groups': {'0': ''}}
print(res['hash']) # {'input': '', 'groups': {'0': ''}}

See the URL Pattern JavaScript documentation for more information.

PSL class

The PSL class allows getting the public suffix and registrable domain of a given host.

First, you need to create a PSL object and load the Public Suffix List. This list can be downloaded from https://publicsuffix.org/list/public_suffix_list.dat. The downloaded file can be loaded using one of the following methods:

1. Use the load function:

   psl = PSL.load('public_suffix_list.dat')
   if (psl is not None):
       print(psl.public_suffix('upa-url.github.io')) # github.io
   

2. Use the PSL constructor:

   try:
       psl = PSL('public_suffix_list.dat')
       # Use psl
   except Exception:
       print('PSL loading error')
   

The Public Suffix List can be loaded from memory using the push interface:

1. Line by line:

   psl = PSL()
   with open('public_suffix_list.dat', 'r', encoding='utf-8') as f:
       for line in f:
           psl.push_line(line.rstrip())
   if psl.finalize():
       # Use psl
   

2. Using the memory buffer, for example, to load a list from the web:

   import urllib.request
   url = 'https://upa-url.github.io/demo/public_suffix_list.dat'
   psl = PSL()
   with urllib.request.urlopen(url) as response:
       while (chunk := response.read(4096)):
           psl.push(chunk)
   if psl.finalize():
       # Use psl
   

The following examples show how to get a public suffix and a registrable domain:

# Get from the host string
print(psl.public_suffix('abc.ålgård.no')) # xn--lgrd-poac.no
print(psl.registrable_domain('abc.ålgård.no')) # abc.xn--lgrd-poac.no

# Get from the host string and do not convert the output to ASCII
print(psl.public_suffix('abc.ålgård.no', ascii=False)) # ålgård.no
print(psl.registrable_domain('abc.ålgård.no', ascii=False)) # abc.ålgård.no

# Get from the URL
url = URL('https://upa-url.github.io/docs/')
print(psl.public_suffix(url)) # github.io
print(psl.registrable_domain(url)) # upa-url.github.io

Conversion functions

First, you need to import library:

import upa_url

Convert from OS file path to URL. The input path must be absolute:

try:
    url = upa_url.url_from_file_path('/c:/path', upa_url.file_path_format.posix)
    print(url.href) # file:///c%3A/path
    url = upa_url.url_from_file_path('c:\\path', upa_url.file_path_format.windows)
    print(url.href) # file:///c:/path
except Exception as err:
    print('Conversion error:', err)

Convert from file URL to OS path:

try:
    url = upa_url.URL('file:///c:/path')
    print(upa_url.path_from_file_url(url, upa_url.file_path_format.posix)) # /c:/path
    print(upa_url.path_from_file_url(url, upa_url.file_path_format.windows)) # c:\path
    url_str = 'file:///c%3A/path'
    print(upa_url.path_from_file_url(url_str, upa_url.file_path_format.posix)) # /c:/path
    print(upa_url.path_from_file_url(url_str, upa_url.file_path_format.windows)) # c:\path
except Exception as err:
    print('Conversion error:', err)

These functions allow the second parameter to be omitted or set to upa_url.file_path_format.native. In that case, the conversion will depend on the operating system on which the script runs.

License

This package is licensed under the BSD 2-Clause License (see LICENSE file).