scitex-web 0.1.0

Web scraping (URLs, images), PubMed search, URL summarization helpers — standalone module from the SciTeX ecosystem

scitex-web

Web scraping + PubMed search + URL summarization helpers, extracted from the SciTeX ecosystem as a standalone package.

Install

pip install scitex-web
pip install "scitex-web[readability]"   # readability-lxml for cleaner extraction

API

import scitex_web as web

# Scraping
web.get_urls(url, pattern=r"\.pdf$")
web.get_image_urls(url, min_size=128)
web.download_images(url, out_dir="imgs", same_domain=True)

# PubMed
web.search_pubmed("CRISPR Cas9 review", retmax=50)

# URL summarization (requires scitex.ai)
web.summarize_url("https://example.com/article")

Status

Standalone fork of scitex.web. Deps: requests / aiohttp / bs4 / tqdm. The umbrella package's scitex.web import path is preserved via a sys.modules-alias bridge.

Decoupling notes:

• scitex.logging.getLogger → stdlib logging.getLogger.
• scitex.str.printc (colored print) → tiny inline ANSI helper.
• scitex.ai.GenAI (used by summarize_url) → deferred import that raises a clear ImportError if the umbrella scitex package isn't installed.

14/23 tests pass (7 pre-existing upstream failures around bs4 mocking that fail in scitex-python too — unrelated to extraction; 2 skipped).

License

AGPL-3.0-only (see LICENSE).