webtoolkit

webtoolkit provides utilities and interfaces for processing and managing Internet data, including URL parsing, HTTP status handling, page type recognition (HTML, RSS, OPML), and support for integrating crawling systems.

Features

URL parsing and cleaning
HTTP status code classification
Page abstraction interfaces (HtmlPage, RssPage, OpmlPage, etc.)
Interfaces for integrating with crawling systems

Remote crawling is supported via crawler-buddy. Provides various crawlers and handlers using interfaces from this package.

Available on pypi.

Install by

pip install webtoolkit

Url processing

To obtain a Url’s data, you can simply do:

url = BaseUrl("https://example.com")

response = url.get_response()

url.get_title()
url.get_description()
url.get_lanugage()
url.get_date_published()
url.get_author()
url.get_feeds()
url.get_entries()

BaseUrl automatically detects and supports many different page types, including YouTube, GitHub, Reddit, and others.

Chain of data

url = BaseUrl("https://example.com")

response = url.get_response()
handler = url.get_handler()
page = handler.get_page()

Page definitions

BaseUrl supports various page types through different classes

HTML pages

page = HtmlPage(url, contents)
page.get_title()
page.get_description()
page.get_lanugage()
page.get_date_published()
page.get_author()
page.get_feeds()

RSS pages

page = RssPage(url, contents)
page.get_title()
page.get_description()
page.get_lanugage()
page.get_date_published()
page.get_author()
page.get_entries()

OPML pages

page = OpmlPage(url, contents)
page.get_entries()

Url location processing

Sanitize link and remove trackers:

location = UrlLocation(link).get_clean()

Extract domain name:

location = UrlLocation(link).get_domain()

Parse and reconstruct links

location = UrlLocation(link)
parsed_data = location.parse_url()
link = location.join(parsed_data) - joins back parsed data into a link

Navigate up the URL structure Go up in the link hierarchy — first to the parent path, then to the domain, and finally to the domain root.

location = UrlLocation(link).up()

is_onion = UrlLocation(link).is_onion()
is_domain = UrlLocation(link).is_domain()
is_image = UrlLocation(link).is_image()
is_audio = UrlLocation(link).is_audio()
is_video = UrlLocation(link).is_video()

is_web_link = UrlLocation(link).is_web_link()          # https://example.com/file.js is a web link
is_webpage_link = UrlLocation(link).is_webpage_link()  # https://example.com/file.js is not a webpage link (it is a file, not a page)

Content processing

Internet contents can be parsed in various ways.

Extracts links from contents

ContentLinkParser(contents).get_links()

Obtain text ready for display

ContentText(text).htmlify()  # returns text, where http links are turned into HTML links
ContentText(text).noattrs()  # removes HTML attributes

Status analysis. Note that from some status we cannot know if page is OK, or not.

is_status_code_valid(status_code)   # provides information if input status code indicates the page is OK
is_status_code_invalid(status_code) # provides information if input status code indicates the page is invalid

HTTP processing - requests

Communication is performed via request - response pairs.

Request HTTP object allows to make HTTP call.

request = PageRequestObject()

To send request to any scraping / crawling server just encode it to GET params

url_data = request_encode(request)

json_data = request_to_json(request)  # json
request = json_to_request(json_data)  # json

HTTP processing - response

Check for valid HTTP responses:

PageResponseObject().is_valid()

Check for invalid HTTP responses:

PageResponseObject().is_invalid()

To check if response is captcha protected

PageResponseObject().is_captcha_protected()

Note: Some status codes may indicate uncertain results (e.g. throttling), where the page cannot be confirmed as valid or invalid yet.

To obtain page structure from response, simply

PageResponseObject().get_page()   # can return HtmlPage, RssPage, etc.

Response communication is done via JSON

json_data = response_to_json(response)    # convert response to JSON
response = json_to_response(json_data)    # convert JSON to response
response_to_file(response, file_name)     # write response to file
response = file_to_response(file_name)    # read response from file

To obtain page contents object:

page = PageResponseObject().get_page()   # returns type of page, be it HtmlPage, RssPage, etc.

Request handling is done simiarly

json_data = request_to_json(request)      # convert request to JSON
response = json_to_request(json_data)     # convert JSON to response
request_to_file(request, file_name)       # write request to file
response = file_to_request(file_name)     # read request from file

Remote interfaces

You can use existing scraping servers.

RemoteUrl - Wrapper around RemoteServer for easy access to remote data. Provides API similar to BaseUrl.

url = RemoteUrl("http://192.168.0.168...")
response = url.get_response()

url.get_title()
url.get_description()
url.get_lanugage()
url.get_date_published()
url.get_author()
url.get_feeds()
url.get_entries()

The communication between client and server should be through JSON requests and responses.

Other classes

RemoteServer - Interface for calling external crawling systems

Standard interfaces

Two standard interfaces

CrawlerInterface - Standard interface for crawler implementations
HandlerInterface - Allows implementing custom handlers for different use cases

Crawlers are different means of obtaining Internet data. Examples: requests, selenium, playwright, httpx, curlcffi. This package does not provide them, to make it more clean and neat.

Handlers are classes that allows automatic deduction of links, places, video codes from links, or data. Examples: youtube handler can use yt-dlp to obtain channel video list, or obtain channel ID, etc.

Default User agents

webtoolkit.get_default_user_agent()

Default User headers

webtoolkit.get_default_headers()

Testing

webtoolkit provides data and facilities that will aid you in testing.

You can use them in your project:

FakeResponse
MockUrl

Project also provides manual tests that check if project works

make tests
make tests-unit # run unit tests
make tests-real # tests performed on real internet data

Release files for webtoolkit 0.1.70

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for webtoolkit 0.1.70
File	Size	Uploaded
webtoolkit-0.1.70.tar.gz	655.3 kB	Sep 25, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for webtoolkit 0.1.70
File	Interpreter	ABI	Platform	Reset
webtoolkit-0.1.70-py3-none-any.whl 678.7 kB Sep 25, 2026	Python 3	none	any	Details

Total release size: 1.3 MB

Release files / webtoolkit-0.1.70.tar.gz

Download URL	webtoolkit-0.1.70.tar.gz
Size	655.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`5a09d6308d427ca7b3c7b043e5b1aceea271fae2343509f1e98295426bdd6d68`
BLAKE2b-256 checksum How to use checksums	`0948fcef404d40c7854a79a03114345a2892d637a3620a9b7530e10bc39edc5c`
Upload date	Sep 25, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/2.1.2 CPython/3.11.2 Linux/6.12.87+rpt-rpi-v8`

Release files / webtoolkit-0.1.70-py3-none-any.whl

Download URL	webtoolkit-0.1.70-py3-none-any.whl
Size	678.7 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`a0d2803d321971e481f694f3914c713832e626db361ec1fa2b46645b4390bcac`
BLAKE2b-256 checksum How to use checksums	`26add0f5ed4b5c8c920501d9f67a60e099b73eaa23ce78ebe4f7065c50a9be8a`
Upload date	Sep 25, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/2.1.2 CPython/3.11.2 Linux/6.12.87+rpt-rpi-v8`