Skip to main content

Extract the top-level domain (TLD) from the URL given.

Project description

Extract the top level domain (TLD) from the URL given. List of TLD names is taken from Public Suffix.

Optionally raises exceptions on non-existing TLDs or silently fails (if fail_silently argument is set to True).

PyPI Version Supported Python versions Build Status MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later Coverage

Prerequisites

  • Python 3.6, 3.7 and 3.8

Support for Python 2.7 and 3.5 is available as well.

Documentation

Documentation is available on Read the Docs.

Installation

Latest stable version on PyPI:

pip install tld

Or latest stable version from GitHub:

pip install https://github.com/barseghyanartur/tld/archive/stable.tar.gz

Usage examples

In addition to examples below, see the jupyter notebook workbook file.

Get the TLD name as string from the URL given

from tld import get_tld

get_tld("http://www.google.co.uk")
# 'co.uk'

get_tld("http://www.google.idontexist", fail_silently=True)
# None

Get the TLD as an object

from tld import get_tld

res = get_tld("http://some.subdomain.google.co.uk", as_object=True)

res
# 'co.uk'

res.subdomain
# 'some.subdomain'

res.domain
# 'google'

res.tld
# 'co.uk'

res.fld
# 'google.co.uk'

res.parsed_url
# SplitResult(
#     scheme='http',
#     netloc='some.subdomain.google.co.uk',
#     path='',
#     query='',
#     fragment=''
# )

Get TLD name, ignoring the missing protocol

from tld import get_tld, get_fld

get_tld("www.google.co.uk", fix_protocol=True)
# 'co.uk'

get_fld("www.google.co.uk", fix_protocol=True)
# 'google.co.uk'

Return TLD parts as tuple

from tld import parse_tld

parse_tld('http://www.google.com')
# 'com', 'google', 'www'

Get the first level domain name as string from the URL given

from tld import get_fld

get_fld("http://www.google.co.uk")
# 'google.co.uk'

get_fld("http://www.google.idontexist", fail_silently=True)
# None

Check if some tld is a valid tld

from tld import is_tld

is_tld('co.uk)
# True

is_tld('uk')
# True

is_tld('tld.doesnotexist')
# False

is_tld('www.google.com')
# False

Update the list of TLD names

To update/sync the tld names with the most recent versions run the following from your terminal:

update-tld-names

Or simply do:

from tld.utils import update_tld_names

update_tld_names()

Note, that this will update all registered TLD source parsers (not only the list of TLD names taken from Mozilla). In order to run the update for a single parser, append uid of that parser as argument.

update-tld-names mozilla

Custom TLD parsers

By default list of TLD names is taken from Mozilla. Parsing implemented in the tld.utils.MozillaTLDSourceParser class. If you want to use another parser, subclass the tld.base.BaseTLDSourceParser, provide uid, source_url, local_path and implement the get_tld_names method. Take the tld.utils.MozillaTLDSourceParser as a good example of such implementation. You could then use get_tld (as well as other tld module functions) as shown below:

from tld import get_tld
from some.module import CustomTLDSourceParser

get_tld(
    "http://www.google.co.uk",
    parser_class=CustomTLDSourceParser
)

Custom list of TLD names

You could maintain your own custom version of the TLD names list (even multiple ones) and use them simultaneously with built in TLD names list.

You would then store them locally and provide a path to it as shown below:

from tld import get_tld
from tld.utils import BaseMozillaTLDSourceParser

class CustomBaseMozillaTLDSourceParser(BaseMozillaTLDSourceParser):

    uid: str = 'custom_mozilla'
    local_path: str = 'tests/res/effective_tld_names_custom.dat.txt'

get_tld(
    "http://www.foreverchild",
    parser_class=CustomBaseMozillaTLDSourceParser
)
# 'foreverchild'

Same goes for first level domain names:

from tld import get_fld

get_fld(
    "http://www.foreverchild",
    parser_class=CustomBaseMozillaTLDSourceParser
)
# 'www.foreverchild'

Note, that in both examples shown above, there the original TLD names file has been modified in the following way:

...
// ===BEGIN ICANN DOMAINS===

// This one actually does not exist, added for testing purposes
foreverchild
...

Free up resources

To free up memory occupied by loading of custom TLD names, use reset_tld_names function with tld_names_local_path parameter.

from tld import get_tld, reset_tld_names

# Get TLD from a custom TLD names parser
get_tld(
    "http://www.foreverchild",
    parser_class=CustomBaseMozillaTLDSourceParser
)

# Free resources occupied by the custom TLD names list
reset_tld_names("tests/res/effective_tld_names_custom.dat.txt")

Support for Python 2.7 and 3.5

As you might have noticed, typing (Python 3.6+) is extensively used in the code. However, Python 3.5 will likely be supported until it’s EOL. All modern recent versions (starting from tld 0.11.7) are fully compatible with Python 2.7 and 3.5 (just works with pip install tld).

Install from pip

pip install tld

Development tips follow:

Python 2.7

Install locally in development mode

python setup.py develop --python-tag py27

Prepare dist

./scripts/prepare_build_py27.sh

Run tests

tox -e py27

Python 3.5

Install locally in development mode

python setup.py develop --python-tag py35

Prepare dist

./scripts/prepare_build_py35.sh

Run tests

tox -e py35

Troubleshooting

If somehow domain names listed here are not recognised, make sure you have the most recent version of TLD names in your virtual environment:

update-tld-names

To update TLD names list for a single parser, specify it as an argument:

update-tld-names mozilla

Testing

Simply type:

./runtests.py

Or use tox:

tox

Or use tox to check specific env:

tox -e py38

Writing documentation

Keep the following hierarchy.

=====
title
=====

header
======

sub-header
----------

sub-sub-header
~~~~~~~~~~~~~~

sub-sub-sub-header
^^^^^^^^^^^^^^^^^^

sub-sub-sub-sub-header
++++++++++++++++++++++

sub-sub-sub-sub-sub-header
**************************

License

MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later

Support

For any issues contact me at the e-mail given in the Author section.

Author

Artur Barseghyan <artur.barseghyan@gmail.com>

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tld-0.12.tar.gz (890.5 kB view details)

Uploaded Source

Built Distributions

tld-0.12-py38-none-any.whl (329.5 kB view details)

Uploaded Python 3.8

tld-0.12-py37-none-any.whl (329.5 kB view details)

Uploaded Python 3.7

tld-0.12-py36-none-any.whl (329.5 kB view details)

Uploaded Python 3.6

tld-0.12-py35-none-any.whl (329.1 kB view details)

Uploaded Python 3.5

tld-0.12-py27-none-any.whl (329.4 kB view details)

Uploaded Python 2.7

File details

Details for the file tld-0.12.tar.gz.

File metadata

  • Download URL: tld-0.12.tar.gz
  • Upload date:
  • Size: 890.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for tld-0.12.tar.gz
Algorithm Hash digest
SHA256 6549aba3be08206eecc552cd2119d8f51525714e98c09848715b2ec4cc99e3d5
MD5 b5b4c4c6dd36eabd14f77d83f32b61db
BLAKE2b-256 fbe91b2ec69441b58455f033facfd0a0540aedc0e10c3c0dd301994d3eedb878

See more details on using hashes here.

File details

Details for the file tld-0.12-py38-none-any.whl.

File metadata

  • Download URL: tld-0.12-py38-none-any.whl
  • Upload date:
  • Size: 329.5 kB
  • Tags: Python 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for tld-0.12-py38-none-any.whl
Algorithm Hash digest
SHA256 824b86fe5fbd5cf85ddd0be04dec5fa8465c63d297ca81ffd1da34c399bb3623
MD5 a73807d2e086df6f4002d96edaf048ef
BLAKE2b-256 d5c4fe8ecfcb56f1afab0da0016b2b25c2001da5bdf3a5f698b06497ae9346f7

See more details on using hashes here.

File details

Details for the file tld-0.12-py37-none-any.whl.

File metadata

  • Download URL: tld-0.12-py37-none-any.whl
  • Upload date:
  • Size: 329.5 kB
  • Tags: Python 3.7
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for tld-0.12-py37-none-any.whl
Algorithm Hash digest
SHA256 b82ea5980ca6c28037d16a6eba512426084ef3d96732adbb371ab7d1711c2040
MD5 f99ef7526c884726edded817572d1f0f
BLAKE2b-256 cb7abcb4ea114f1f375fa82b3f9bbfd206be9ba511bf2f9ab6c6f8cbec5d0168

See more details on using hashes here.

File details

Details for the file tld-0.12-py36-none-any.whl.

File metadata

  • Download URL: tld-0.12-py36-none-any.whl
  • Upload date:
  • Size: 329.5 kB
  • Tags: Python 3.6
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for tld-0.12-py36-none-any.whl
Algorithm Hash digest
SHA256 6d1c2db9d0dd50b1eafe7521a7999eb097eecd6b8a5f5a10245568ab0e0c97d2
MD5 59ddade76da2490e4410a350f990ed4e
BLAKE2b-256 e78bc5213430150aa8bbf85affea0b8818aa111bd85f94a3ceba5f49bd6fb3f9

See more details on using hashes here.

File details

Details for the file tld-0.12-py35-none-any.whl.

File metadata

  • Download URL: tld-0.12-py35-none-any.whl
  • Upload date:
  • Size: 329.1 kB
  • Tags: Python 3.5
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for tld-0.12-py35-none-any.whl
Algorithm Hash digest
SHA256 dcaeac28e8d42ef3cbe717a4f9ba3a9a49e50b25dad0d4dd669a44e91bb02186
MD5 c3914cc83a522f2aa9f18ce6c5aecd74
BLAKE2b-256 c35c6843e26ef6ad688ffec935c638f03db45c194c4250151a379a3d73c7cb92

See more details on using hashes here.

File details

Details for the file tld-0.12-py27-none-any.whl.

File metadata

  • Download URL: tld-0.12-py27-none-any.whl
  • Upload date:
  • Size: 329.4 kB
  • Tags: Python 2.7
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.6.9

File hashes

Hashes for tld-0.12-py27-none-any.whl
Algorithm Hash digest
SHA256 cf886df97b70a24b1c43cea6fca78e42091e697ea0420caf0c707f8f6826b426
MD5 7f60c15f8c75fc98abcd2e029bb9ff46
BLAKE2b-256 b87b2e797aaaf29d0acd923523aba2850cddbd88e0f0951a9353b50e522a5adb

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page