web-scout-ai

Web research that actually reads the sources.

web-scout-ai is an agentic Python research pipeline that searches the web, opens and extracts the sources it finds, follows up when evidence is incomplete, and returns a cited answer together with a full audit trail of what it could — and could not — read.

Give it a question or a URL:

from web_scout import run_web_research

result = await run_web_research(
    "Indonesia biodiversity hotspots current status and recent trend"
)

print(result.synthesis)
print(f"Read {len(result.scraped)} sources across {len(result.queries)} searches")

Instead of treating search snippets as evidence, web-scout-ai tries to read the underlying source itself before allowing it to support the final answer.

What it actually does

web-scout-ai is an async Python research pipeline, not a search-result wrapper. It searches, opens the selected sources, routes each response to the right extractor, evaluates whether the collected evidence is sufficient, and synthesizes only from sources it successfully read.

The browser fallback can pass many JS-gated and Cloudflare-protected pages, but it is not a guarantee: sites can still block automation. Those URLs are reported in result.bot_detected instead of disappearing or being presented as evidence. Use the package only where you have permission and in accordance with the source site's terms.

Quick start

1. Install

pip install web-scout-ai
web-scout-setup

web-scout-setup installs the Patchright-managed Chromium browser and its system dependencies. It may request sudo for OS-level browser libraries.

2. Configure keys

The default research models run on Amazon Bedrock Mantle. Vision fallback still uses Gemini, and open-web discovery uses Serper:

export AWS_BEARER_TOKEN_BEDROCK="your-bedrock-api-key"
export GEMINI_API_KEY="your-gemini-api-key"
export SERPER_API_KEY="your-serper-api-key"

BEDROCK_MANTLE_API_KEY is accepted as an alias for the bearer token. OpenAI models on Mantle, including GPT-5.6 and GPT-6 Luna, use LiteLLM's route in us-east-1.

To use the Exa search backend instead (search_backend="exa"):

export EXA_API_KEY="your-exa-api-key"

Direct-URL mode does not use a search backend, so it only needs the API key for your configured model provider.

3. Run research

import asyncio

from web_scout import run_web_research


async def main():
    result = await run_web_research(
        query="What are the main threats to coral reefs worldwide?",
        cache=True,
    )

    print(result.synthesis)

    print("\nSources read:")
    for source in result.scraped:
        print(f"- {source.title or source.url}: {source.url}")

    if result.bot_detected or result.scrape_failed:
        print(
            f"\nCould not read "
            f"{len(result.bot_detected) + len(result.scrape_failed)} source(s)"
        )


asyncio.run(main())

Three ways to use it

Open-web research

Generate several searches, read the strongest results in parallel, evaluate coverage, and search again if important evidence is missing.

result = await run_web_research(
    query="What is driving the global adaptation finance gap?",
)

Domain-restricted research

Keep discovery and hub deepening focused on authoritative domains.

result = await run_web_research(
    query="Latest evidence on sea-level rise",
    include_domains=["ipcc.ch", "nasa.gov"],
)

Direct URL extraction

Skip search and start from a page, document, API endpoint, image, or database listing.

result = await run_web_research(
    query="Extract the recommended adaptation measures and supporting evidence",
    direct_url="https://example.org/report.pdf",
)

For a document, the pipeline reads that document without wandering into site navigation. For a list or database page, it can rank and follow relevant records and their linked primary documents.

The return value is an audit trail

run_web_research() returns a typed WebResearchResult:

class WebResearchResult(BaseModel):
    synthesis: str
    scraped: list[UrlEntry]
    scrape_failed: list[UrlEntry]
    blocked_by_policy: list[UrlEntry]
    source_http_error: list[UrlEntry]
    scraped_irrelevant: list[UrlEntry]
    bot_detected: list[UrlEntry]
    snippet_only: list[UrlEntry]
    queries: list[SearchQuery]

UrlEntry contains url, title, and content. SearchQuery contains query, num_results_returned, and domains_restricted.

The synthesizer is instructed to use scraped evidence only. A deterministic final check rejects citations to invented or snippet-only URLs and retries the synthesis with feedback.

Research depth

# Faster default
await run_web_research(query="...", research_depth="standard")

# More searches and sources, with a stricter coverage threshold
await run_web_research(query="...", research_depth="deep")

The coverage evaluator can stop early when the evidence already answers the question. You can add your own acceptance conditions with coverage_criteria:

result = await run_web_research(
    query="Compare national methane policies",
    coverage_criteria="Include at least one primary government source per country.",
)

Custom extractor guidance

Use extractor_guidance to refine what each per-source extractor keeps and how it organizes evidence. The guidance applies only to extraction; it does not alter search query generation, coverage evaluation, or final synthesis.

result = await run_web_research(
    query="Tunisia agricultural trends",
    extractor_guidance="""
Country of interest: Tunisia

- Keep findings explicitly about Tunisia.
- Also keep regional findings whose stated scope includes Tunisia, even when the
  source does not name Tunisia; label them with the source's scope, such as
  "North Africa:".
- Discard purely global findings and findings about geographies that do not
  include Tunisia.
""",
)

Guidance augments rather than replaces WebScout's extractor instructions. The base contract always takes precedence for source grounding, tool constraints, output fields, page-type handling, and the exact no-evidence sentinel.

Configuration

The defaults are Bedrock Mantle GPT-6 Luna for research and extraction, TypeSafe Jev for follow-up link selection, and gemini/gemini-3.7-flash for vision. Model IDs follow LiteLLM provider naming, so each stage can use a different provider or model. Set WEB_SCOUT_FOLLOWUP_BACKEND=gpt to use the Agents SDK model in models["followup_selector"] instead of Jev. Jev reads TYPESAFE_API_KEY.

models = {
    # Used as the fallback for query generation, coverage, and synthesis
    "web_researcher": "openai/gpt-4o-mini",

    # Reads and extracts individual sources
    "content_extractor": "gemini/gemini-2.0-flash",

    # Optional stage-specific choices
    "query_generator": "openai/gpt-4o-mini",
    "coverage_evaluator": "openai/gpt-4o-mini",
    "synthesiser": "openai/gpt-4o-mini",
    "followup_selector": "openai/gpt-4o-mini",  # used when WEB_SCOUT_FOLLOWUP_BACKEND=gpt
    "vision_fallback": "gemini/gemini-2.0-flash",
}

result = await run_web_research(query="...", models=models)
# Or keep Luna for link selection:
# Set WEB_SCOUT_FOLLOWUP_BACKEND=gpt in the environment, then pass models as usual.

Provider credentials are read from their standard environment variables, such as OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, TYPESAFE_API_KEY, or AWS credentials for Bedrock.

Public API

Fetch one URL and get a FetchResult. This uses the same Scrapling fetcher as the research pipeline (fast HTTP, then a stealth browser when the page looks blocked or JavaScript-rendered):

from web_scout import PDF_MAGIC_BYTES, fetch

result = await fetch("https://example.org/report.pdf")
page = await fetch("https://example.org/article", wait_for="article")

result.body is the raw response bytes for binary payloads such as PDFs and images. HTML and other text responses leave body as None and put the decoded page in result.html_content. Failures do not raise: result.error is set when the URL is rejected or the fetch fails, and result.status is outside the 2xx range for HTTP errors. Pass exclude_domains to skip hosts before any network call.

PDF_MAGIC_BYTES is b"%PDF", the four-byte header every PDF file starts with. Check the body against it when you need to know whether the fetch returned a PDF. That is more reliable than result.content_type, which download URLs often report as application/octet-stream:

is_pdf = bool(result.body and result.body.startswith(PDF_MAGIC_BYTES))
if is_pdf:
    pdf_bytes = result.body

fetch_pdf is a thin wrapper around fetch that does this check and returns the PDF bytes. It raises RuntimeError when the fetch fails or the body does not start with PDF_MAGIC_BYTES:

from web_scout import fetch_pdf

pdf_bytes = await fetch_pdf("https://openknowledge.fao.org/bitstreams/c1caede2-ea98-46b0-b663-4cae429e05d3/download")

result = await run_web_research(
    query="latest IPCC findings on sea-level rise",
    models=None,                       # optional; GPT-6 Luna defaults, Gemini vision
    search_backend="serper",          # "serper" (Google via serper.dev) or "exa" (exa.ai)
    research_depth="standard",        # "standard", "deep", or a custom dict
    include_domains=["ipcc.ch"],       # optional discovery restriction
    direct_url=None,                   # optional; skips search when set
    domain_expertise="climate science",  # optional prompt context
    exclude_domains=None,              # optional blocklist; nothing is blocked
                                       # by default. See "Domain policy" below.
    max_pdf_pages=50,                  # pages converted from each PDF
    max_content_chars=30_000,          # characters passed to the extractor per source
    cache=False,                       # accepted; URL and PDF caches stay on
    cache_storage=None,               # Prefect block, e.g. a GCS bucket
    refresh_pdf_cache=False,          # recompute cached PDF parses
    refresh_url_cache=False,          # recompute cached URL fetches
    coverage_criteria=None,            # extra evidence requirements
    extractor_guidance=None,           # optional per-source extraction guidance
)

Domain policy

Nothing is blocked by default: exclude_domains=None means every domain is eligible for search and scraping. To block domains, pass a list explicitly. A curated list of platforms that waste the scrape budget (social/video sites with no scrapeable text, bot-blocked search engines, hard-paywalled publishers) is exported as RECOMMENDED_EXCLUDE_DOMAINS:

from web_scout import run_web_research, RECOMMENDED_EXCLUDE_DOMAINS

# Opt in to the curated list
result = await run_web_research(
    query="...",
    exclude_domains=RECOMMENDED_EXCLUDE_DOMAINS,
)

# Curated list plus your own additions
result = await run_web_research(
    query="...",
    exclude_domains=[*RECOMMENDED_EXCLUDE_DOMAINS, "statista.com"],
)

Exclusions are applied natively by search backends that support them (Exa's excludeDomains); backends without native support (Serper) rely on the post-search URL filter, which always runs. Contradictions fail fast: a domain appearing in both include_domains and exclude_domains, or a direct_url whose host is excluded, raises ValueError.

Source caching

URL fetches and PDF parses are cached with Prefect. A second request for the same URL reuses the stored page or document. A second request for the same PDF bytes reuses the parse even when the URL is different. Query-specific extraction and synthesis still run each time.

By default the cache is on disk at ~/.cache/web-scout and survives a new process. Pass cache_storage to store both caches in a caller-supplied Prefect block, such as a GCS bucket:

from prefect_gcp.cloud_storage import GcsBucket

result = await run_web_research(
    query="...",
    cache_storage=GcsBucket(bucket="my-cache"),
    refresh_pdf_cache=False,
    refresh_url_cache=False,
)

refresh_pdf_cache=True and refresh_url_cache=True recompute those entries. cache=True is still accepted and does not turn the cache off. Failed scrapes, final answers, query-specific summaries, and click-driven browser sessions are not cached. Concurrent workers wait rather than fetching or parsing the same URL or PDF at the same time.

Pipeline, in one view

question
   │
   ├─ generate targeted searches ─ search in parallel ─ select diverse URLs
   │                                                     │
   │                                                     ▼
   │      static HTML ────────┐                    fetch in parallel
   │      JS / bot challenge ─┤                          │
   │      documents ──────────┼─ route + extract ◀───────┘
   │      JSON / images ──────┘          │
   │                                    ▼
   └─ follow-up search ◀── evaluate evidence coverage
                                        │ sufficient
                                        ▼
                              grounded synthesis
                                        │
                                        ▼
                               citation validation

Where it fits

Use web-scout-ai when your application needs the contents of real pages and documents—not only search snippets—and you want the source successes and failures returned as structured data.

It is intentionally a bounded research component. If you only need search links, use a search API directly. If you need an open-ended autonomous research process with human checkpoints, put this package inside a broader agent workflow.

Requirements

• Python 3.13
• An API key for the configured LLM provider
• A Serper or Exa API key for search mode
• Chromium setup for rendered pages, interactive sites, and browser fallbacks

Classification and resource reuse

Semantic classification defaults to TypeSafe Jev and requires TYPESAFE_API_KEY. Set WEB_SCOUT_CLASSIFICATION_BACKEND=gpt to use the existing GPT classifiers instead. Query generation, factual extraction, visual descriptions and synthesis continue to use your configured generative models. Feature-specific switches and the global override are described below.

CPU PDF pipelines initialize on the first document and are reused. Browser and HTTP resources are reused on the same event loop; after requests finish, call await web_scout.scraping.close_resources() at application shutdown. See the performance report for measurements, limits, benchmark commands and compatibility details.

Contributing

The main extension point is SearchBackend. New backends should implement the async search() contract and return normalized results (title, url, snippet).

Bug reports and focused pull requests are welcome at github.com/RSO9192/web-scout-ai.

License

MIT

Jev / GPT selection

Set DISABLE_JEV=true to use the retained GPT implementations for every Jev-backed feature. It overrides all feature settings and does not require TYPESAFE_API_KEY. Unset or DISABLE_JEV=false preserves Jev defaults. Values are case-insensitive and surrounding whitespace is ignored.

Each feature defaults to jev. The existing WEB_SCOUT_CLASSIFICATION_BACKEND remains a shared default for features without a specific setting. Invalid backend values raise an error; missing Jev credentials remain an error when an active feature selects Jev. GPT paths use the existing configured models and provider credentials; disabling Jev does not change model IDs or switch Bedrock models to the OpenAI endpoint.

The followup_backend argument has been removed from run_web_research(). Replace followup_backend="luna" with WEB_SCOUT_FOLLOWUP_BACKEND=gpt, or followup_backend="jev" with WEB_SCOUT_FOLLOWUP_BACKEND=jev.

Orchestrator link selection

Orchestrator uses DefaultCrawler, which selects links from already-parsed content using the same Jev/GPT selectors as the research pipeline. Select the backend with WEB_SCOUT_CRAWLER_BACKEND=jev|gpt; DISABLE_JEV=true forces GPT. WEB_SCOUT_FOLLOWUP_BACKEND independently controls follow-ups in run_web_research().

Crawl4AICrawler remains an alias for DefaultCrawler to preserve existing imports. llm_config=None still requests heuristic-only selection. Existing config objects with provider, api_token and base_url are accepted for custom GPT connections; third-party extraction-strategy options are no longer used. No Crawl4AI package, browser or setup is required.