Skip to main content

Abstract WebTools

Composable, manager-based utilities for fetching, parsing, crawling, and mirroring web content.

Abstract WebTools wraps the messy parts of web access — HTTP sessions, TLS/cipher configuration, user‑agent rotation, retries, HTML parsing, link extraction, crawling, headless browsers, and media downloading — behind a set of small, composable managers. The managers share a single URL → request → soup pipeline, so a page is fetched once and reused everywhere downstream instead of being re‑fetched by every layer.


Table of contents


Why

Most scraping code re‑implements the same plumbing on every project: building a session, picking a user agent, tuning TLS so a server doesn't reject you, handling retries, parsing HTML, then doing it all again for the next step.

Abstract WebTools factors each concern into a manager and threads shared instances through the chain. Pass an existing req_mgr (or source code) into any higher‑level manager and it is reused as‑is — no rebuild, no second network request.


Install

pip install abstract_webtools

Optional extras:

pip install "abstract_webtools[drivers]"   # selenium + webdriver-manager
pip install "abstract_webtools[media]"     # yt-dlp + m3u8 for video downloads
pip install "abstract_webtools[gui]"       # PyQt/PySimpleGUI helpers

Core runtime deps: requests, urllib3, beautifulsoup4. Browser and media features pull in selenium / playwright / yt-dlp as needed.


Quick start

from abstract_webtools import get_soup, get_source, linkManager

# Fetch + parse a page (one request, reused internally)
soup = get_soup("https://example.com")
print(soup.title.text)

# Just the raw HTML
html = get_source("https://example.com")

# All links + image links on the page
lm = linkManager("https://example.com")
print(lm.all_desired_links)
print(lm.all_desired_image_links)

Architecture: the manager chain

The core managers form a layered pipeline. Each layer accepts the layer(s) below it and reuses them when provided:

urlManager        normalize / validate / vary URLs
   └─ requestManager   sessions, retries, TLS, UA  ── networkManager ┐
        └─ soupManager       BeautifulSoup parsing                   ├─ userAgentManager
             ├─ linkManager       link / image extraction            ├─ cipherManager
             └─ crawlManager       site crawling / sitemaps          └─ sslManager + tlsAdapter

Every layer has a matching factory function that detects and reuses an existing instance:

Factory Returns Reuses when given
get_url_mgr(url=, url_mgr=) urlManager url_mgr
get_req_mgr(url=, url_mgr=, source_code=, req_mgr=) requestManager req_mgr
get_source(...) HTML string source_code / req_mgr
get_soup_mgr(...) soupManager soup_mgr / req_mgr
get_soup(...) BeautifulSoup soup / soup_mgr / source_code
get_crawl_mgr(...) crawlManager req_mgr / url_mgr
get_managed_session(...) requests.Session req_mgr

Because every factory short‑circuits on an instance you pass in, the whole chain is built once and shared:

from abstract_webtools import get_req_mgr, get_soup_mgr, linkManager

req = get_req_mgr("https://example.com")        # fetches once
soup_mgr = get_soup_mgr(req_mgr=req)            # no re-fetch
links = linkManager(req_mgr=req)                # no re-fetch

The managers

Manager Responsibility
urlManager Parse, validate, normalize and generate URL variants.
requestManager requests.Session with retries, timeouts, TLS adapter, UA, proxies, cookies; optional Selenium fallback.
networkManager Mounts the TLS adapter and wires proxies/cookies/UA into the session.
userAgentManager Realistic user agents and per‑URL headers (random or pinned by OS/browser).
cipherManager Cipher‑suite strings for TLS.
sslManager / tlsAdapter SSL context + HTTPAdapter for fine‑grained TLS control.
soupManager BeautifulSoup parsing, meta/link extraction, attribute discovery.
linkManager Internal/image link extraction with desired/undesired filters.
crawlManager Recursive crawling, sitemap generation, domain link discovery.
middleManager UnifiedWebManager — one lazy facade over the whole chain.
usurpManager Full‑site mirror: pages + assets + styles, references rewritten for offline use.
videoDownloader Video/media download via yt-dlp / m3u8, wired to the managed session/UA.
seleneumManager / playwriteManager Headless‑browser source fetching for JS‑rendered pages.

Common recipes

Get a page's source / soup

from abstract_webtools import get_source, get_soup, get_soup_mgr

html = get_source("https://example.com")
soup = get_soup("https://example.com")

# Reuse already-fetched HTML — no network call
soup2 = get_soup(source_code=html)

# Soup manager exposes parsing helpers
sm = get_soup_mgr("https://example.com")
print(sm.get_all_attribute_values(tags_list=["a", "img"]))
from abstract_webtools import linkManager

lm = linkManager(
    "https://example.com",
    link_attr_value_desired=["/blog/"],      # keep only links containing this
    image_link_tags="img",
)
print(lm.all_desired_links)
print(lm.find_all_domain())                  # unique domains found

Crawl a site

from abstract_webtools import get_crawl_mgr, get_domain_crawl

crawl = get_crawl_mgr("https://example.com")
domain_links = get_domain_crawl("https://example.com", max_depth=3)

One shared context: UnifiedWebManager

UnifiedWebManager lazily builds and caches url_mgr, req_mgr, source_code, soup_mgr, soup, plus link_mgr / crawl_mgr — all over a single fetch.

from abstract_webtools import UnifiedWebManager

web = UnifiedWebManager("https://example.com")
web.url_mgr      # built on demand
web.source_code  # fetched once
web.soup         # parsed once
web.link_mgr     # shares the same chain — no re-fetch
web.crawl_mgr

# Or start from HTML you already have (zero network):
web = UnifiedWebManager(source_code="<html>...</html>")
web.soup.title

A managed requests.Session

Need a plain session, but configured with a real user agent, ciphers, the TLS adapter and proxies? Ask the stack for one — it never fetches just to build it, and reuses an existing req_mgr's session when given:

from abstract_webtools import get_managed_session

session = get_managed_session(user_agent="MyBot/1.0")
resp = session.get("https://example.com")

Mirror an entire site (usurpManager)

usurpManager saves a working offline copy of a site — pages and styles intact. By default it recursively captures the whole site: every same‑domain page link and all referenced media. It follows CSS url(...) / @import (including @font-face and cross‑domain CDN fonts), handles srcset, inline style="" and <style> blocks, downloads scripts/images/linked files, and rewrites every reference to a relative local path so the result renders straight from file://.

from abstract_webtools import usurpit

# Full recursive capture of the entire site (unlimited depth by default):
result = usurpit("https://example.com", output_dir="example_mirror")
print(result["output_dir"], len(result["pages"]), "pages")

Or drive it directly for more control:

from abstract_webtools import usurpManager, get_req_mgr

req = get_req_mgr("https://example.com")
site = usurpManager(
    "https://example.com",
    req_mgr=req,                      # reuse the managed session
    output_dir="example_mirror",
    max_depth=None,                   # default: unlimited (whole site); set an int to cap
    mirror_external_assets=True,      # pull CDN css/fonts so styles work (default)
)
summary = site.main()
  • The crawl is breadth‑first and unlimited‑depth by default (max_depth=None); the visited‑set keeps it finite/loop‑free. Pass an integer max_depth to bound it.
  • Pages are mirrored within the origin host; referenced assets may come from CDNs (set mirror_external_assets=False to stay strictly on‑origin).
  • A single url → local path map keeps references consistent and shared assets are fetched exactly once.
  • For heavily JS‑rendered sites, fetch the rendered HTML first via seleneumManager / playwriteManager.

Download video / media

from abstract_webtools import get_video_info, downloadvideo

info = get_video_info("https://www.youtube.com/watch?v=...")   # metadata only
downloadvideo("https://www.youtube.com/watch?v=...", download_directory="videos")

The downloader pulls its user agent (and proxy) from the shared request stack and threads them into yt-dlp, so downloads use the same identity as the rest of your scrape. You can inject an existing req_mgr / ua_mgr:

from abstract_webtools import VideoDownloader, get_req_mgr

req = get_req_mgr("https://example.com")
VideoDownloader(url="https://example.com/video.mp4", req_mgr=req,
                download_directory="videos")

Agent tools (agentManager)

agentManager exposes the manager chain as JSON-safe tools for LLM agents. Every tool takes plain arguments, returns a dict and never raises; failures come back as {"error": {"code", "message"}}. A fetched page is cached for five minutes, so chained calls on one URL don't refetch.

Engines. Every page tool takes engine= (and wait_for= a CSS selector for browser engines). auto uses HTTP first and a browser only when the page is a client-rendered shell; web_engines() lists what works on the host.

Engine Built on
requests requestManager over a per-host stack: userAgentManager identity, networkManager session (sslManager/cipherManager TLS adapter, cookies, retries), DynamicRateLimiter
selenium seleneumManager (Chrome binary/service resolution, throwaway profile)
playwright imitationBrowser.PlaywrightEngine (falls back to system Chrome if the bundled build is missing)
stealth imitationBrowser.SeleniumEngine with stealth mode
curl_cffi / cloudscraper / httpx optional (pip install abstract_webtools[scrape]): browser TLS impersonation, Cloudflare challenge, HTTP/2

Add your own with register_engine(MyEngine(), auto="http"|"browser") — an engine implements fetch(url, *, timeout, max_bytes, wait_for, headers) (and optionally screenshot).

Tool Purpose
web_fetch raw body + status, headers, redirects (JSON parsed)
web_read readable structured text, paged with offset / next_offset
web_grep find a phrase/regex on a page → snippets with offsets
web_meta title, description, canonical, OpenGraph, JSON-LD, outline, counts
web_links links with anchor text, internal/external, emails/phones
web_find / web_extract elements or named fields via soupManager's forgiving finder
web_forms / web_tables / web_media forms, tables, images/video/audio/documents/feeds
web_sitemap / web_crawl robots + sitemap discovery; bounded same-site BFS crawl
web_url / web_status parse a URL (no network); HEAD probe with redirect chain
web_screenshot page image through any browser engine
web_video provider detection + yt-dlp metadata/download via videoDownloader
web_download / web_mirror save a file; offline site copy via usurpManager
web_engines / web_clear_cache engine availability; drop cached pages (and site identities)

AGENT_WEB_TOOLS maps names to functions and web_tool(action, args) dispatches one by name.

from abstract_webtools import web_read, web_extract, web_screenshot
page = web_read("https://example.com", max_chars=4000)
spa  = web_read("https://app.example.com", engine="playwright", wait_for="#content")
data = web_extract("https://example.com", {"heading": "h1", "links": "a"})
shot = web_screenshot("https://example.com", engine="selenium", full_page=True)

The page store

Every page the agent tools fetch is kept in a SQLite file, ~/.local/share/abstract_webtools/pages.sqlite3 (ABSTRACT_WEBTOOLS_STORE names another file; off turns the store off). A page is saved whole and as sections: its own blocks (a heading block, a list item or card, a run of table rows), each with the heading it sits under and a CSS selector back to its element.

Tool What it does
web_sections(url, offset, limit, query) Read a page block by block, paged; query keeps matching blocks
web_saved_search(query, url, host) Full-text search over every saved page, without fetching
web_saved(host) List saved pages, the room each site takes, and the store's size
web_forget(url, host, older_than_days, everything) Remove saved pages

Nothing is removed on its own: web_saved shows what has built up and web_forget clears it. A refetch with unchanged markup adds nothing; changed markup is kept as a new snapshot. The five-minute fetch cache (CACHE_TTL) is separate: it only decides when a tool fetches again.

Design notes

  • Reuse over rebuild. Every factory and constructor honors an instance you pass in. Supplying source_code or a req_mgr means zero extra network requests downstream.
  • One session, fully configured. TLS ciphers, SSL context, the HTTP adapter, user agent, proxies and cookies are assembled once by the request stack and reused — including by usurpManager and videoDownloader.
  • Optional heavy deps stay optional. Browser/media/GUI extras are imported defensively so the core package imports without them.

Testing

The repo ships dependency‑light regression tests (only requests + beautifulsoup4 required) that load the real modules under a controlled namespace and assert the no‑refetch behavior and the site mirror:

python tests/test_manager_chain.py        # url/request/soup/link chain reuse
python tests/test_video_usurp_chain.py    # managed session for video + usurp
python tests/test_usurp_mirror.py         # full-site mirror with styles

Contributing

Issues and PRs welcome at AbstractEndeavors/abstract_webtools. Please keep new functionality threaded through the shared manager chain (accept and reuse url_mgr / req_mgr / source_code) rather than re‑fetching, and add a dependency‑light test where practical.

Metadata

Release files for abstract-webtools 0.1.6.440

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for abstract-webtools 0.1.6.440
File Size Uploaded Provenance Actions
abstract_webtools-0.1.6.440.tar.gz 179.9 kB Without provenance Details for abstract_webtools-0.1.6.440.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for abstract-webtools 0.1.6.440
File Interpreter ABI Platform Actions
abstract_webtools-0.1.6.440-py3-none-any.whl Python 3 none any Details for abstract_webtools-0.1.6.440-py3-none-any.whl

Total release size: 410.7 kB

Release files / abstract_webtools-0.1.6.440.tar.gz

Download URL abstract_webtools-0.1.6.440.tar.gz
Size 179.9 kB
Tags Source
SHA-256 checksum
How to use checksums
5486c31cd2ad4da2d3b7edf6f67d0bf120a5a06864ef03cddb03039c8018203f
BLAKE2b-256 checksum
How to use checksums
8847c73da0c6fafb10a5e6e1147c1cba75dc28479a492a6415cfd810f455de56
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / abstract_webtools-0.1.6.440-py3-none-any.whl

Download URL abstract_webtools-0.1.6.440-py3-none-any.whl
Size 230.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f0735d4d9adb934f764b9c1141642cdb17de7c6c95638ed41528c17222b5fa07
BLAKE2b-256 checksum
How to use checksums
23e323ab69d56b479151ee7b7ea51e35d536141438bcab095813934516914b02
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for abstract-webtools 0.1.6.440

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.6.440 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page