OSS Research Web (experimental)

Standalone Python package. Search (SearXNGSearch) and extraction (PageReader) are independently usable; ResearchSession adds per-agent references/open/find. No Modis, Codex, model provider, API key, or paid fallback is required. SearXNG is an external service, not bundled Python code.

This is an experimental prerelease. Install with pip install oss-research-web==0.3.0a1; deployment still requires a SearXNG service. See v0.3 image validation, v0.2 validation, initial validation, and validation/ACCEPTANCE.md for evidence. The current candidate uses the keyless Google and Yahoo SearXNG adapters, not Google's paid API. Engine availability is environment-dependent.

Development

uv sync --extra dev --extra browser
uv run pytest
uv run ruff check .
uv run python -m build
uv run playwright install chromium

Linux is currently required for the parser process resource limits. The tested interpreter was Python 3.13.9. A supplied browser executable can also be selected. PDF screenshots additionally require Poppler's pdftoppm executable on PATH (for example, the poppler-utils OS package). Search and text extraction work without Poppler; rendering reports renderer_unavailable if it is absent.

Version 0.3.0a1 adds page figures, image search, bounded decoding, original-pixel crops, explicit previews, PDF crops and multimodal input content to the existing filtered search and source-snapshot workflows. See image usage. It does not enable this provider in deployed agents.

Reproduce the live evaluation

From this package directory (not the oss-agent deployment directory):

docker compose -f deploy/compose.yaml up -d
uv run python validation/run.py --output validation/runs/CHOOSE-A-NEW-RUN-NAME
uv run python validation/features.py --output validation/runs/CHOOSE-ANOTHER-NEW-RUN-NAME
docker compose -f deploy/compose.yaml down

The runner refuses an existing output directory. It makes 30 sequential searches, 20 different searches with four concurrent callers, and ten independent page reads. Requests are paced to one start/second across a shared provider. This launches no model and consumes no Valyu credits. Do not reuse the old run names.

For the browser fixture tests, set RUN_BROWSER_TESTS=1; also set PLAYWRIGHT_BROWSERS_PATH if Chromium was installed in a non-default location.

The dependency lock is for reproducible validation. Browser installation is optional for HTTP/HTML/PDF use. Run browser work in a container when appropriate.

API

import asyncio
from research_web import SearXNGSearch, PageReader, ResearchSession, SearchFilters


async def main():
    async with SearXNGSearch("http://127.0.0.1:18991") as search, PageReader() as reader:
        session = ResearchSession(search, reader)
        results = await session.search(
            "continual learning reservoir replay", limit=5,
            filters=SearchFilters(allowed_domains=("arxiv.org", "proceedings.neurips.cc")),
        )
        page = await session.open(results.results[0].url)
        print(page.view(start=1, lines=80))
        print(session.find("replay"))
        print(session.sources())


asyncio.run(main())

SearchResponse distinguishes successful empty results, degraded engines, and exceptions. WebError.code is machine readable. Sources carry stable URL-derived IDs. Pages carry a content hash, final URL, fetch timestamp and extraction method. Line slices are one-based. Find is literal text, with bounded matches/context; untrusted regex is deliberately not executed.

An async event sink can receive started/completed/failed/cache_hit events with operation IDs. A future Modis adapter can translate these without provider-specific logic. Cache entries are immutable and bounded by count, bytes and TTL. Session references/current page are never shared between agents.

HTTP extraction keeps main/article content when available, strips navigation and scripts, preserves code/tables/links, and resolves relative links. Browser fallback is optional and only used for sparse HTML; it is not CAPTCHA or paywall bypass. PDF extraction is text-only, page-labelled, with no OCR or layout accuracy promise. Optional PDF screenshots preserve visual layout for inspection; they do not transcribe it or make the model using this library capable of vision.

Research navigation and provenance

See the workflow contract for the full API and limits. Page.source_id identifies a URL; Page.snapshot_id identifies a particular extracted representation and original-byte hash. open(snapshot_id) reuses that exact retained version. open(url, refresh=True) fetches a new version. An expired snapshot fails explicitly rather than silently refetching a possibly changed page. sources() distinguishes search-only discoveries from opened, hashed documents. links(snapshot_id) lists numbered links; click(id, target=snapshot_id) follows one. find(text, target=snapshot_id) searches an earlier document without changing the current page. view() returns bounded excerpts and continuation positions, including an offset when one line exceeds the character budget. Full retained text remains available; a view limit never discards the remainder of a document.

paper = await session.open("https://arxiv.org/pdf/1706.03762")
print(paper.pdf_text(start_page=3, end_page=4))
image = await session.screenshot(paper.snapshot_id, page=3, max_edge=1600)
# image.png is a PNG byte string; image.data_url() is suitable for a vision input.
# PDF page numbers and link IDs are one-based throughout this package.

For image-only PDFs, text extraction still reports no_text. Use pdf_info(url) and screenshot(url, page=...) to inspect them visually. OCR and automatic model/gateway tool integration remain separate work. General image search and HTML image extraction are now available through the explicit image APIs.

Public page requests reject credentials, local/private destinations and non-HTTP schemes; DNS is resolved and pinned per request, with redirects revalidated. Only GET is supported. Browser subresources use the same fetcher; service workers and WebSockets are blocked. SearXNG's configured local endpoint is explicitly trusted. The prototype is a library, not a multi-tenant security boundary. Use a restricted container for browser execution and untrusted document parsing in production.

Validation uses a separate SearXNG container; no agent profile or active gateway is modified. Publishing and integration remain gated by the independent evidence.