liburlparser 2.1.0

Fastest domain extractor library written in C++ with python binding.

First and complete library for parsing url in C++ and Python and Command Line

About The Project

liburlparser is a powerful domain extractor library written in C++ with Python bindings. It provides efficient URL parsing capabilities for both C++ and Python, making it a valuable tool for projects that involve working with web addresses.

Features

Here are some key features of liburlparser:

1. Multiple Language Support:

   • liburlparser can be used in multiple programming languages, including Python, C++, and Shell.
   • It offers an intuitive interface that remains consistent across both C++ and Python.

2. Clean Code Design:

   • The library provides separate classes for each concept: Url, and Hostname/IPv4/IPv6 for whatever a URL's host actually is (RFC 3986 defines a host as either a domain name or an IP address, never both - liburlparser mirrors that directly instead of forcing IP addresses through domain-name-shaped fields).
   • This separation allows for cleaner and more organized code when dealing with URLs.

3. Public Suffix List Support:

   • liburlparser supports known combinatorial suffixes (e.g., "ac.ir") using the public_suffix_list.
   • It can also handle unknown suffixes (e.g., "comm" in "google.comm").

4. Automatic Public Suffix List Updates:

   • Before each build and deployment, liburlparser updates the public_suffix_list automatically.

5. Hostname Properties:

   • The Hostname class includes properties such as subdomain, domain, domain name, and suffix.

6. IPv4 / IPv6 Support:

   • IPv4/IPv6 addresses used as a URL's host are recognized and parsed as such - not run through domain-name logic. Both support integer conversion and arithmetic (+, -, +=, -=, increment/decrement).

7. URL Properties:

   • The Url class provides properties like protocol, userinfo, host (classified as Hostname/IPv4/IPv6), port, path, query parameters, and fragment.

Usage

Command Line

python -m liburlparser --help # show help section
python -m liburlparser --version # show version
python -m liburlparser --url "https://mail.google.com/about" | jq #return as json
python -m liburlparser --host "mail.google.com" | jq # return as json

Python

you can use liburlparser so intutively

all of classes has help section

import liburlparser
help(liburlparser)
print(liburlparser.__version__)

from liburlparser import Url, Hostname, IPv4, IPv6
help(Url)
help(Hostname)

parse url and host

from liburlparser import Url, Hostname, Host
## parse url:
url = Url("https://ee.aut.ac.ir/#id") # parse all part of url
print(url, url.host_text, url.fragment, url.host, url.to_dict(), url.to_json())
## url.host is a Hostname, IPv4, or IPv6 object - whichever this URL's host
## actually is (an IP address has no "domain"/"suffix", so it's simply not
## forced through Hostname at all)
host = url.host  # a Hostname, here: ee.aut.ac.ir
# or
host = Hostname("ee.aut.ac.ir")
# or
host = Hostname.from_url("https://ee.aut.ac.ir/#id") # the fastest way for parsing host from url
print(host, host.domain, host.suffix, host.to_dict(), host.to_json())

## for a host you don't know in advance is a domain or an IP, use Host():
host2 = Host("192.168.1.1")   # is_ipv4() -> True
host3 = Host("ee.aut.ac.ir")  # is_hostname() -> True
print(host2.is_ipv4(), host2, int(host2.get_ipv4()))          # True 192.168.1.1 3232235777
print(host3.is_hostname(), host3, host3.get_hostname().domain) # True ee.aut.ac.ir aut

IPv4/IPv6 addresses support integer conversion and arithmetic:

from liburlparser import IPv4, IPv6

ip = IPv4("192.168.1.1")
print(int(ip))          # 3232235777
print(ip + 1)            # 192.168.1.2
print(ip - IPv4("192.168.1.0"))  # 1 (distance between two addresses)

ip6 = IPv6("2001:db8::1")
print(ip6.high64, ip6.low64)
print(ip6 + 1)            # 2001:db8::2

Also there is some helping api to get better performance for some small tasks

# if you need to extract the host of url as a string without any parsing
host_str = Url.extract_host("https://ee.aut.ac.ir/about") # very fast

C++

there is some examples in examples folder

#include "urlparser.h"
...
/// for parsing url
urlparser::url url("https://ee.aut.ac.ir/about");
std::string_view host_text = url.host_text(); // the raw host text, unclassified

/// url.host() is a urlparser::host (hostname/ipv4/ipv6, whichever this
/// URL's host actually is)
if (auto* h = url.host().try_hostname()) {
    std::cout << h->domain() << "." << h->suffix();
}

/// for parsing a host you already have (or a full URL) directly:
urlparser::hostname host("ee.aut.ac.ir");
// or
urlparser::hostname host2 = urlparser::hostname::from_url("https://ee.aut.ac.ir/about");
// or, when you don't know in advance whether it's a domain or an IP:
urlparser::host classified = urlparser::host::from_url("http://192.168.1.1/about"); // -> is_ipv4() true

you can see all methods in python we can use in c++ very easily

Installation

C++

Requires a C++17 compiler and CMake >= 3.19. The library is a static library plus one header (urlparser.h), and its CMake target is urlparser::urlparser - the same name no matter how you get it:

1. FetchContent

include(FetchContent)
FetchContent_Declare(
  liburlparser
  GIT_REPOSITORY https://github.com/mohammadraziei/liburlparser.git
  GIT_TAG        master
)
FetchContent_MakeAvailable(liburlparser)

target_link_libraries(my_app PRIVATE urlparser::urlparser)

2. From the pip package

The Python wheel also ships the compiled C++ library, its header and a CMake package config (same idea as python -m nanobind --cmake_dir), so a C++ project can use a plain pip install liburlparser without building anything:

find_package(Python3 REQUIRED COMPONENTS Interpreter)
execute_process(
    COMMAND ${Python3_EXECUTABLE} -m liburlparser --cmake-dir
    OUTPUT_VARIABLE liburlparser_DIR
    OUTPUT_STRIP_TRAILING_WHITESPACE
    COMMAND_ERROR_IS_FATAL ANY)

find_package(liburlparser CONFIG REQUIRED)
target_link_libraries(my_app PRIVATE urlparser::urlparser)

For build systems that prefer plain paths over the target, find_package also sets:

The same locations are available from the command line and from Python:

python -m liburlparser --cmake-dir     # directory with liburlparserConfig.cmake
python -m liburlparser --include-dir   # directory with urlparser.h
python -m liburlparser --lib-dir       # directory with the compiled library

import liburlparser
liburlparser.get_cmake_dir(), liburlparser.get_include_dir(), liburlparser.get_lib_dir()

A complete example project is in examples/find_liburlparser_via_python.

3. Build and install from source

git clone https://github.com/mohammadraziei/liburlparser
cd liburlparser

# configure and build (-B creates the 'build' directory)
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build

# run the example (on Windows with multi-config generators: ./build/Release/example.exe)
./build/example

# install the library, header and CMake package (add --prefix <dir> to choose where)
cmake --install build

Then, in your project (add -DCMAKE_PREFIX_PATH=<prefix> if you used a custom prefix):

find_package(liburlparser CONFIG REQUIRED)
target_link_libraries(my_app PRIVATE urlparser::urlparser)

Python and Command Line:

Be aware that it requires python>=3.10

Installation

from source

git clone https://github.com/mohammadraziei/liburlparser
cd liburlparser

pip install .

# Optional: If you want to see build logs
pip install . -v

pip by pypi

pip install liburlparser

Or

pip by git

pip install git+https://github.com/mohammadraziei/liburlparser

Or

manually

git clone https://github.com/mohammadraziei/liburlparser
pip install ./liburlparser

Performance

Up-to-date, reproducible results for both C++ and Python (compared against other libraries) are on the Benchmarks page. The report is generated locally and published with the documentation:

cmake -S benchmarks -B build-bench -DCMAKE_BUILD_TYPE=Release
cmake --build build-bench --target urlparser_benchmarks_report
# -> benchmarks/results/report.html

The two tables below are the original measurements from March 2022.

Extract From Host

Tests were run on a file containing 10 million random domains from various top-level domains (Mar. 13rd 2022)

Extract From URL

The test was conducted on a file containing 1 million random urls (Mar. 13rd 2022)

License

Distributed under the MIT License. See LICENSE for more information.

Stats

Contact

Project Link:

• https://github.com/mohammadraziei/liburlparser
• https://pypi.org/project/liburlparser
• Documentation and Benchmarks