pydomainextractor 0.2.2

Highly optimized Domain Name Extraction library written in C++

Highly optimized domain name extraction library written in C++

Table of Contents

• Table of Contents
• About The Project
  • Built With
  • Performance
  • Prerequisites
  • Installation
• Usage
• License
• Contact

About The Project

PyDomainExtractor is a library intended for parsing domain names into their parts fast. The library is written in C++ to achieve the highest performance possible.

Built With

• GNU libidn2
• Public Suffix List

Performance

Test was measured on a file containing 10 million random domains from various TLDs

Library	Function	Time
tldextract	__call__	67.0s
publicsuffix2	publicsuffix2.get_tld	25.8s
PyDomainExtractor	pydomainextractor.extract	2.76s

Prerequisites

In order to compile this package you should have GCC, libidn2, and Python development package installed.

• Fedora

sudo dnf install python3-devel libidn2-devel gcc-c++

• Ubuntu 18.04

sudo apt install python3-dev libidn2-dev g++-8

Installation

pip3 install PyDomainExtractor

Usage

The usual use case:

import pydomainextractor


# Loads the current supplied version of PublicSuffixList from the repository. Does not download any data.
pydomainextractor.load()

pydomainextractor.extract('google.com')
>>> {
>>>     'subdomain': '',
>>>     'domain': 'google',
>>>     'suffix': 'com'
>>> }

# Loads a custom SuffixList data. Should follow PublicSuffixList's format.
pydomainextractor.load(
    'tld\n'
    'custom.tld\n'
)

pydomainextractor.extract('google.com')
>>> {
>>>     'subdomain': 'google',
>>>     'domain': 'com',
>>>     'suffix': ''
>>> }

pydomainextractor.extract('google.custom.tld')
>>> {
>>>     'subdomain': '',
>>>     'domain': 'google',
>>>     'suffix': 'custom.tld'
>>> }

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Gal Ben David - [e-mail]

Project Link: https://github.com/wavenator/PyDomainExtractor