scitex-web

Web scraping + PubMed search + URL summarization helpers.

Full Documentation · uv pip install scitex-web[all]

Problem and Solution

Architecture

scitex_web/
├── _scrape.py          # get_urls / get_image_urls / download_images
├── _pubmed.py          # search_pubmed (E-utilities client)
├── _summarize.py       # summarize_url (deferred GenAI import)
└── _utils.py           # shared logging + ANSI helpers

flowchart LR
    URL[URL] -->|get_urls / get_image_urls| L[Filtered links]
    L -->|download_images| F[Local files]
    Q[Query string] -->|search_pubmed| R[Parsed PubMed records]
    A[Article URL] -->|summarize_url| S[Summary text]
    S -.uses.-> AI[(scitex.ai.GenAI<br/>deferred import)]
    style F fill:#27ae60,stroke:#2c3e50,color:#fff
    style R fill:#27ae60,stroke:#2c3e50,color:#fff
    style S fill:#27ae60,stroke:#2c3e50,color:#fff

_{Figure 1. Module layout. Three independent capabilities — scraping, PubMed search, URL summarization. Only summarization needs the umbrella scitex for the GenAI client.}

Installation

pip install scitex-web
pip install "scitex-web[readability]"   # readability-lxml for cleaner extraction

Quick Start

import scitex_web as web

results = web.search_pubmed("CRISPR Cas9 review", retmax=5)
images = web.get_image_urls("https://example.com/gallery", min_size=128)

1 Interfaces

import scitex_web as web

# Scraping
web.get_urls(url, pattern=r"\.pdf$")
web.get_image_urls(url, min_size=128)
web.download_images(url, out_dir="imgs", same_domain=True)

# PubMed
web.search_pubmed("CRISPR Cas9 review", retmax=50)

# URL summarization (requires scitex.ai umbrella)
web.summarize_url("https://example.com/article")

Demo

import scitex_web as web

# 1) Scrape PDFs from a faculty page
pdfs = web.get_urls("https://lab.example.edu/publications", pattern=r"\.pdf$")
#   → ['https://lab.example.edu/papers/2024_smith.pdf', ...]

# 2) Bulk-download figures (same domain, ≥256px)
web.download_images(
    "https://example.com/gallery",
    out_dir="imgs",
    same_domain=True,
    min_size=256,
)

# 3) Search PubMed
records = web.search_pubmed("CRISPR Cas9 review", retmax=5)
for r in records:
    print(r["pmid"], r["title"])

# 4) Summarize an article (requires umbrella scitex)
print(web.summarize_url("https://example.com/article"))

flowchart TD
    Start[Start] --> Want{Goal?}
    Want -- "harvest URLs" --> A[get_urls / get_image_urls]
    Want -- "save images" --> B[download_images]
    Want -- "search literature" --> C[search_pubmed]
    Want -- "summarize article" --> D[summarize_url]
    A --> Out[List / DataFrame]
    B --> Out2[Files on disk]
    C --> Out3[Parsed records]
    D --> Out4[Text summary]
    style Out  fill:#27ae60,stroke:#2c3e50,color:#fff
    style Out2 fill:#27ae60,stroke:#2c3e50,color:#fff
    style Out3 fill:#27ae60,stroke:#2c3e50,color:#fff
    style Out4 fill:#27ae60,stroke:#2c3e50,color:#fff

_{Figure 2. Demo. Pick a helper by output shape: links, files, records, or summary text.}

Status

Standalone fork of scitex.web. Deps: requests / aiohttp / bs4 / tqdm. The umbrella package's scitex.web import path is preserved via a sys.modules-alias bridge.

Decoupling notes:

• scitex.logging.getLogger → stdlib logging.getLogger.
• scitex.str.printc (colored print) → tiny inline ANSI helper.
• scitex.ai.GenAI (used by summarize_url) → deferred import that raises a clear ImportError if the umbrella scitex package isn't installed.

Part of SciTeX

scitex-web is part of SciTeX. Install via the umbrella with pip install scitex[web] to use as scitex.web (Python) or scitex web ... (CLI).

Four Freedoms for Research

0. The freedom to run your research anywhere — your machine, your terms.
1. The freedom to study how every step works — from raw data to final manuscript.
2. The freedom to redistribute your workflows, not just your papers.
3. The freedom to modify any module and share improvements with the community.

AGPL-3.0 — because we believe research infrastructure deserves the same freedoms as the software it runs on.

License

AGPL-3.0-only (see LICENSE).