Skip to main content
courlan

coURLan: Filter, clean, and sample URLs for web crawling

Python package Python versions Code Coverage Documentation

Why coURLan?

"It is important for the crawler to visit 'important' pages first, so that the fraction of the Web that is visited (and kept up to date) is more meaningful." (Cho et al. 1998)

"Given that the bandwidth for conducting crawls is neither infinite nor free, it is becoming essential to crawl the Web in not only a scalable, but efficient way, if some reasonable measure of quality or freshness is to be maintained." (Edwards et al. 2001)

This library provides an additional "brain" for web crawling, scraping and document management. It facilitates web navigation through a set of filters, enhancing the quality of resulting document collections:

  • Validate, clean, and normalize URLs, including tracker removal
  • Save bandwidth by steering clear of spam, unwanted content types, and low-value pages
  • Identify pages by language with locale-aware heuristics
  • Sample URLs by domain and pinpoint pages relevant for link gathering
  • Store, deduplicate, and schedule URLs with a crawl-ready data store
  • Use it from Python or on the command line

Crawl or Skip? Filter and Sample URLs with courlan (video)

Wade through the Web and keep the good links. The logo shows a courlan, also called limpkin.

Installation

This package requires Python 3.10 or higher and is tested on Linux, macOS and Windows systems.

courlan is available on PyPI and installs with pip:

pip install courlan
pip install git+https://github.com/adbar/courlan.git  # latest available code

For older Python versions, see the installation guide.

Quickstart

In Python — validate, clean, and filter URLs with check_url:

>>> from courlan import check_url

# cleaned URL and domain name
>>> check_url('https://example.org/page?utm_source=twitter')
('https://example.org/page', 'example.org')

# rejected URLs return None
>>> check_url('http://666.0.0.1/')  # bogus domain

# language-aware filtering
>>> check_url('https://www.un.org/en/about-us', language='en')
('https://www.un.org/en/about-us', 'un.org')
>>> check_url('https://www.un.org/en/about-us', language='de')  # returns None

Most filters revolve around the strict and language arguments. See the check_url guide for all options.

On the command line — filter a file of URLs:

# one URL per line in urls.txt
courlan -i urls.txt -o cleaned.txt -d discarded.txt
# cleaned.txt contains accepted URLs, discarded.txt contains rejected ones

Run courlan --help or see the CLI documentation for all command-line options.

Python

Sampling by domain name

>>> from courlan import sample_urls
>>> my_urls = ['https://example.org/' + str(x) for x in range(100)]
>>> my_sample = sample_urls(my_urls, 10)

See the sampling guide for all options.

Web crawling and URL handling

Use extract_links() for general-purpose link extraction. For crawl-aware extraction with robots.txt rules and link prioritization, use filter_links() instead — it returns two lists: regular links and priority (navigation) links.

>>> from courlan import extract_links
>>> doc = '<html><body><a href="test/link.html">Link</a></body></html>'
>>> extract_links(doc, "https://example.org")
['https://example.org/test/link.html']

For frontier management utilities (is_external, is_navigation_page, is_not_crawlable, …) see the crawling guide.

Python helpers

>>> from courlan import clean_url
>>> clean_url('HTTPS://WWW.DWDS.DE:443/')
'https://www.dwds.de'

For normalize_url, validate_url, get_base_url, get_hostinfo, and other utilities see the cleaning guide.

UrlStore class

The UrlStore class allows for storing and retrieving domain-classified URLs, where a URL like https://example.org/path/testpage is stored as the path /path/testpage within the domain https://example.org:

>>> from courlan import UrlStore
>>> store = UrlStore()
>>> store.add_urls(['https://example.org/page1', 'https://example.org/page2'])
>>> store.get_url('https://example.org')
'https://example.org/page1'
>>> store.find_unvisited_urls('https://example.org')
['https://example.org/page2']

See the UrlStore guide for all methods, options, persistence, and crawl scheduling.

Settings

Language filtering reads language codes in URL paths for any language, and in subdomains with strict=True. Language names in lang= query values are mapped for English and German by default. See the settings reference for how to review and override filtering rules.

Author

Initially launched to create text databases for research purposes at the Berlin-Brandenburg Academy of Sciences (DWDS and ZDL units), this package continues to be maintained but its future development depends on community support.

If you value this software or depend on it for your product, consider sponsoring it and contributing to its codebase. Your support on GitHub or ko-fi.com will help maintain and enhance this package. Visit the Contributing page for more information.

Reach out via the software repository or the contact page for inquiries, collaborations, or feedback.

For more on Courlan's software ecosystem see this graphic.

License

coURLan is distributed under the Apache 2.0 license.

Versions prior to v1 were under GPLv3+ license.

Similar work

These Python libraries perform URL handling and normalization but do not provide language-aware filtering, content heuristics, crawl scheduling, or a domain-classified URL store:

References

  • Cho, J., Garcia-Molina, H., & Page, L. (1998). Efficient crawling through URL ordering. Computer networks and ISDN systems, 30(1-7), 161–172.
  • Edwards, J., McCurley, K. S., and Tomlin, J. A. (2001). "An adaptive model for optimizing performance of an incremental web crawler". In Proceedings of the 10th international conference on World Wide Web - WWW'01, pp. 106–113.

Metadata

Release files for courlan 1.5.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for courlan 1.5.0
File Size Uploaded Provenance Actions
courlan-1.5.0.tar.gz 108.3 kB Without provenance Details for courlan-1.5.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for courlan 1.5.0
File Interpreter ABI Platform Actions
courlan-1.5.0-py3-none-any.whl Python 3 none any Details for courlan-1.5.0-py3-none-any.whl

Total release size: 173.2 kB

Release files / courlan-1.5.0.tar.gz

Download URL courlan-1.5.0.tar.gz
Size 108.3 kB
Tags Source
SHA-256 checksum
How to use checksums
68adfdef9eed04240ce081d4fc9faf796db0e820795e9cc545b822aed8d26fc0
BLAKE2b-256 checksum
How to use checksums
7b2424d78cee217be9435ebc246009524f0372d1aa86b01714d06e753671a0f1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.1

Release files / courlan-1.5.0-py3-none-any.whl

Download URL courlan-1.5.0-py3-none-any.whl
Size 64.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a5b121409f864e6ddf39a026ebafabf59c64511d11b336ae30ad4034bbd42b77
BLAKE2b-256 checksum
How to use checksums
8b594f564893812e0b1f2775437f43bb6b9d492e36c659dd3a1f52a29a0b189e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.1

Security for courlan 1.5.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.5.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page