Extract the top-level domain (TLD) from the URL given.
Project description
Extract the top level domain (TLD) from the URL given. List of TLD names is taken from Mozilla.
Optionally raises exceptions on non-existing TLDs or silently fails (if fail_silently argument is set to True).
Documentation
Documentation is available on Read the Docs.
Installation
Latest stable version on PyPI:
pip install tld
Or latest stable version from GitHub:
pip install https://github.com/barseghyanartur/tld/archive/stable.tar.gz
Or latest stable version from BitBucket:
pip install https://bitbucket.org/barseghyanartur/tld/get/stable.tar.gz
Usage examples
In addition to examples below, see the jupyter notebook workbook file.
Get the TLD name as string from the URL given
from tld import get_tld get_tld("http://www.google.co.uk") # 'co.uk' get_tld("http://www.google.idontexist", fail_silently=True) # None
Get the TLD as an object
from tld import get_tld res = get_tld("http://some.subdomain.google.co.uk", as_object=True) res # 'co.uk' res.subdomain # 'some.subdomain' res.domain # 'google' res.tld # 'co.uk' res.fld # 'google.co.uk' res.parsed_url # SplitResult( # scheme='http', # netloc='some.subdomain.google.co.uk', # path='', # query='', # fragment='' # )
Get TLD name, ignoring the missing protocol
from tld import get_tld, get_fld get_tld("www.google.co.uk", fix_protocol=True) # 'co.uk' get_fld("www.google.co.uk", fix_protocol=True) # 'google.co.uk'
Return TLD parts as tuple
from tld import parse_tld parse_tld('http://www.google.com') # 'com', 'google', 'www'
Get the first level domain name as string from the URL given
from tld import get_fld get_fld("http://www.google.co.uk") # 'google.co.uk' get_fld("http://www.google.idontexist", fail_silently=True) # None
Check if some tld is a valid tld
from tld import is_tld is_tld('co.uk) # True is_tld('uk') # True is_tld('tld.doesnotexist') # False is_tld('www.google.com') # False
Update the list of TLD names
To update/sync the tld names with the most recent versions run the following from your terminal:
update-tld-names
Or simply do:
from tld.utils import update_tld_names update_tld_names()
Note, that this will update all registered TLD source parsers (not only the list of TLD names taken from Mozilla). In order to run the update for a single parser, append uid of that parser as argument.
update-tld-names mozilla
Custom TLD parsers
By default list of TLD names is taken from Mozilla. Parsing implemented in the tld.utils.MozillaTLDSourceParser class. If you want to use another parser, subclass the tld.base.BaseTLDSourceParser, provide uid, source_url, local_path and implement the get_tld_names method. Take the tld.utils.MozillaTLDSourceParser as a good example of such implementation. You could then use get_tld (as well as other tld module functions) as shown below:
from tld import get_tld from some.module import CustomTLDSourceParser get_tld( "http://www.google.co.uk", parser_class=CustomTLDSourceParser )
Custom list of TLD names
You could maintain your own custom version of the TLD names list (even multiple ones) and use them simultaneously with built in TLD names list.
You would then store them locally and provide a path to it as shown below:
from tld import get_tld from tld.utils import BaseMozillaTLDSourceParser class CustomBaseMozillaTLDSourceParser(BaseMozillaTLDSourceParser): uid: str = 'custom_mozilla' local_path: str = 'tests/res/effective_tld_names_custom.dat.txt' get_tld( "http://www.foreverchild", parser_class=CustomBaseMozillaTLDSourceParser ) # 'foreverchild'
Same goes for first level domain names:
from tld import get_fld get_fld( "http://www.foreverchild", parser_class=CustomBaseMozillaTLDSourceParser ) # 'www.foreverchild'
Note, that in both examples shown above, there the original TLD names file has been modified in the following way:
... // ===BEGIN ICANN DOMAINS=== // This one actually does not exist, added for testing purposes foreverchild ...
Free up resources
To free up memory occupied by loading of custom TLD names, use reset_tld_names function with tld_names_local_path parameter.
from tld import get_tld, reset_tld_names # Get TLD from a custom TLD names parser get_tld( "http://www.foreverchild", parser_class=CustomBaseMozillaTLDSourceParser ) # Free resources occupied by the custom TLD names list reset_tld_names("tests/res/effective_tld_names_custom.dat.txt")
Support for Python 2.7 and 3.5
As you might have noticed, typing (Python 3.6+) is extensively used in the code. However, Python 3.5 will likely be supported until it’s EOL. All modern recent versions (starting from tld 0.11.7) are fully compatible with Python 2.7 and 3.5 (just works with pip install tld).
Install from pip
pip install tld
Development tips follow:
Python 2.7
Install locally in development mode
python setup.py develop --python-tag py27
Prepare dist
./scripts/prepare_build_py27.sh
Run tests
./runtests.py src_py27/ --python-tag py27
Python 3.5
Install locally in development mode
python setup.py develop --python-tag py35
Prepare dist
./scripts/prepare_build_py35.sh
Run tests
./runtests.py src_py35/ --python-tag py35
Troubleshooting
If somehow domain names listed here are not recognised, make sure you have the most recent version of TLD names in your virtual environment:
update-tld-names
To update TLD names list for a single parser, specify it as an argument:
update-tld-names mozilla
Testing
Simply type:
./runtests.py
Or use tox:
tox
Or use tox to check specific env:
tox -e py36
Writing documentation
Keep the following hierarchy.
===== title ===== header ====== sub-header ---------- sub-sub-header ~~~~~~~~~~~~~~ sub-sub-sub-header ^^^^^^^^^^^^^^^^^^ sub-sub-sub-sub-header ++++++++++++++++++++++ sub-sub-sub-sub-sub-header **************************
License
MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later
Support
For any issues contact me at the e-mail given in the Author section.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Filename, size | File type | Python version | Upload date | Hashes |
---|---|---|---|---|
Filename, size tld-0.11.10-py27-none-any.whl (328.8 kB) | File type Wheel | Python version py27 | Upload date | Hashes View |
Filename, size tld-0.11.10-py35-none-any.whl (328.6 kB) | File type Wheel | Python version py35 | Upload date | Hashes View |
Filename, size tld-0.11.10-py36-none-any.whl (328.9 kB) | File type Wheel | Python version py36 | Upload date | Hashes View |
Filename, size tld-0.11.10-py37-none-any.whl (328.9 kB) | File type Wheel | Python version py37 | Upload date | Hashes View |
Filename, size tld-0.11.10-py38-none-any.whl (328.9 kB) | File type Wheel | Python version py38 | Upload date | Hashes View |
Filename, size tld-0.11.10.tar.gz (888.9 kB) | File type Source | Python version None | Upload date | Hashes View |