web-scout-ai

Web research that actually reads the sources.

web-scout-ai is an agentic Python research pipeline that searches the web, opens and extracts the sources it finds, follows up when evidence is incomplete, and returns a cited answer together with a full audit trail of what it could — and could not — read.

Give it a question or a URL:

from web_scout import run_web_research

result = await run_web_research(
    "Indonesia biodiversity hotspots current status and recent trend"
)

print(result.synthesis)
print(f"Read {len(result.scraped)} sources across {len(result.queries)} searches")

Instead of treating search snippets as evidence, web-scout-ai tries to read the underlying source itself before allowing it to support the final answer.

What it actually does

web-scout-ai is an async Python research pipeline, not a search-result wrapper. It searches, opens the selected sources, routes each response to the right extractor, evaluates whether the collected evidence is sufficient, and synthesizes only from sources it successfully read.

The browser fallback can pass many JS-gated and Cloudflare-protected pages, but it is not a guarantee: sites can still block automation. Those URLs are reported in result.bot_detected instead of disappearing or being presented as evidence. Use the package only where you have permission and in accordance with the source site's terms.

Quick start

1. Install

pip install web-scout-ai
web-scout-setup

web-scout-setup installs the Patchright-managed Chromium browser and its system dependencies. It may request sudo for OS-level browser libraries.

2. Configure keys

The default research models run on Amazon Bedrock Mantle. Vision fallback still uses Gemini, and open-web discovery uses Serper:

export AWS_BEARER_TOKEN_BEDROCK="your-bedrock-api-key"
export GEMINI_API_KEY="your-gemini-api-key"
export SERPER_API_KEY="your-serper-api-key"

BEDROCK_MANTLE_API_KEY is accepted as an alias for the bearer token. OpenAI models on Mantle, including GPT-5.6 and GPT-6 Luna, use LiteLLM's route in us-east-1.

To use the Exa search backend instead (search_backend="exa"):

export EXA_API_KEY="your-exa-api-key"

Direct-URL mode does not use a search backend, so it only needs the API key for your configured model provider.

3. Run research

import asyncio

from web_scout import run_web_research


async def main():
    result = await run_web_research(
        query="What are the main threats to coral reefs worldwide?",
        cache=True,
    )

    print(result.synthesis)

    print("\nSources read:")
    for source in result.scraped:
        print(f"- {source.title or source.url}: {source.url}")

    if result.bot_detected or result.scrape_failed:
        print(
            f"\nCould not read "
            f"{len(result.bot_detected) + len(result.scrape_failed)} source(s)"
        )


asyncio.run(main())

Three ways to use it

Open-web research

Generate several searches, read the strongest results in parallel, evaluate coverage, and search again if important evidence is missing.

result = await run_web_research(
    query="What is driving the global adaptation finance gap?",
)

Domain-restricted research

Keep discovery and hub deepening focused on authoritative domains.

result = await run_web_research(
    query="Latest evidence on sea-level rise",
    include_domains=["ipcc.ch", "nasa.gov"],
)

Direct URL extraction

Skip search and start from a page, document, API endpoint, image, or database listing.

result = await run_web_research(
    query="Extract the recommended adaptation measures and supporting evidence",
    direct_url="https://example.org/report.pdf",
)

For a document, the pipeline reads that document without wandering into site navigation. For a list or database page, it can rank and follow relevant records and their linked primary documents.

The return value is an audit trail

run_web_research() returns a typed WebResearchResult:

class WebResearchResult(BaseModel):
    synthesis: str
    scraped: list[UrlEntry]
    scrape_failed: list[UrlEntry]
    blocked_by_policy: list[UrlEntry]
    source_http_error: list[UrlEntry]
    scraped_irrelevant: list[UrlEntry]
    bot_detected: list[UrlEntry]
    snippet_only: list[UrlEntry]
    queries: list[SearchQuery]

UrlEntry contains url, title, and content. SearchQuery contains query, num_results_returned, and domains_restricted.

The synthesizer is instructed to use scraped evidence only. A deterministic final check rejects citations to invented or snippet-only URLs and retries the synthesis with feedback.

Research depth

# Faster default
await run_web_research(query="...", research_depth="standard")

# More searches and sources, with a stricter coverage threshold
await run_web_research(query="...", research_depth="deep")

The coverage evaluator can stop early when the evidence already answers the question. You can add your own acceptance conditions with coverage_criteria:

result = await run_web_research(
    query="Compare national methane policies",
    coverage_criteria="Include at least one primary government source per country.",
)

Custom extractor guidance

Use extractor_guidance to refine what each per-source extractor keeps and how it organizes evidence. The guidance applies only to extraction; it does not alter search query generation, coverage evaluation, or final synthesis.

result = await run_web_research(
    query="Tunisia agricultural trends",
    extractor_guidance="""
Country of interest: Tunisia

- Keep findings explicitly about Tunisia.
- Also keep regional findings whose stated scope includes Tunisia, even when the
  source does not name Tunisia; label them with the source's scope, such as
  "North Africa:".
- Discard purely global findings and findings about geographies that do not
  include Tunisia.
""",
)

Guidance augments rather than replaces WebScout's extractor instructions. The base contract always takes precedence for source grounding, tool constraints, output fields, page-type handling, and the exact no-evidence sentinel.

Configuration

The defaults are Bedrock Mantle GPT-6 Luna for research and extraction, TypeSafe Jev for follow-up link selection, and gemini/gemini-3.7-flash for vision. Model IDs follow LiteLLM provider naming, so each stage can use a different provider or model. Pass followup_backend="luna" to use the Agents SDK model in models["followup_selector"] instead of Jev. Jev reads TYPESAFE_API_KEY.

models = {
    # Used as the fallback for query generation, coverage, and synthesis
    "web_researcher": "openai/gpt-4o-mini",

    # Reads and extracts individual sources
    "content_extractor": "gemini/gemini-2.0-flash",

    # Optional stage-specific choices
    "query_generator": "openai/gpt-4o-mini",
    "coverage_evaluator": "openai/gpt-4o-mini",
    "synthesiser": "openai/gpt-4o-mini",
    "followup_selector": "openai/gpt-4o-mini",  # used when followup_backend="luna"
    "vision_fallback": "gemini/gemini-2.0-flash",
}

result = await run_web_research(query="...", models=models)
# Or keep Luna for link selection:
# result = await run_web_research(query="...", followup_backend="luna", models=models)

Provider credentials are read from their standard environment variables, such as OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, TYPESAFE_API_KEY, or AWS credentials for Bedrock.

Public API

Fetch one URL and get a FetchResult. This uses the same Scrapling fetcher as the research pipeline (fast HTTP, then a stealth browser when the page looks blocked or JavaScript-rendered):

from web_scout import PDF_MAGIC_BYTES, fetch

result = await fetch("https://example.org/report.pdf")
page = await fetch("https://example.org/article", wait_for="article")

result.body is the raw response bytes for binary payloads such as PDFs and images. HTML and other text responses leave body as None and put the decoded page in result.html_content. Failures do not raise: result.error is set when the URL is rejected or the fetch fails, and result.status is outside the 2xx range for HTTP errors. Pass exclude_domains to skip hosts before any network call.

PDF_MAGIC_BYTES is b"%PDF", the four-byte header every PDF file starts with. Check the body against it when you need to know whether the fetch returned a PDF. That is more reliable than result.content_type, which download URLs often report as application/octet-stream:

is_pdf = bool(result.body and result.body.startswith(PDF_MAGIC_BYTES))
if is_pdf:
    pdf_bytes = result.body

fetch_pdf is a thin wrapper around fetch that does this check and returns the PDF bytes. It raises RuntimeError when the fetch fails or the body does not start with PDF_MAGIC_BYTES:

from web_scout import fetch_pdf

pdf_bytes = await fetch_pdf("https://example.org/report.pdf")

result = await run_web_research(
    query="latest IPCC findings on sea-level rise",
    models=None,                       # optional; GPT-6 Luna defaults, Gemini vision
    search_backend="serper",          # "serper" (Google via serper.dev) or "exa" (exa.ai)
    research_depth="standard",        # "standard", "deep", or a custom dict
    include_domains=["ipcc.ch"],       # optional discovery restriction
    direct_url=None,                   # optional; skips search when set
    domain_expertise="climate science",  # optional prompt context
    exclude_domains=None,              # optional blocklist; nothing is blocked
                                       # by default. See "Domain policy" below.
    max_pdf_pages=50,                  # pages converted from each PDF
    max_content_chars=30_000,          # characters passed to the extractor per source
    cache=False,                       # process-local source cache
    coverage_criteria=None,            # extra evidence requirements
    extractor_guidance=None,           # optional per-source extraction guidance
)

Domain policy

Nothing is blocked by default: exclude_domains=None means every domain is eligible for search and scraping. To block domains, pass a list explicitly. A curated list of platforms that waste the scrape budget (social/video sites with no scrapeable text, bot-blocked search engines, hard-paywalled publishers) is exported as RECOMMENDED_EXCLUDE_DOMAINS:

from web_scout import run_web_research, RECOMMENDED_EXCLUDE_DOMAINS

# Opt in to the curated list
result = await run_web_research(
    query="...",
    exclude_domains=RECOMMENDED_EXCLUDE_DOMAINS,
)

# Curated list plus your own additions
result = await run_web_research(
    query="...",
    exclude_domains=[*RECOMMENDED_EXCLUDE_DOMAINS, "statista.com"],
)

Exclusions are applied natively by search backends that support them (Exa's excludeDomains); backends without native support (Serper) rely on the post-search URL filter, which always runs. Contradictions fail fast: a domain appearing in both include_domains and exclude_domains, or a direct_url whose host is excluded, raises ValueError.

Source caching

With cache=True, successful raw source artifacts are reused by later run_web_research() calls in the same Python process. Pages and documents are not fetched or converted again, but query-specific extraction and synthesis still run each time.

The cache is in-memory only. Failed scrapes, final answers, query-specific summaries, and click-driven browser sessions are not cached.

Pipeline, in one view

question
   │
   ├─ generate targeted searches ─ search in parallel ─ select diverse URLs
   │                                                     │
   │                                                     ▼
   │      static HTML ────────┐                    fetch in parallel
   │      JS / bot challenge ─┤                          │
   │      documents ──────────┼─ route + extract ◀───────┘
   │      JSON / images ──────┘          │
   │                                    ▼
   └─ follow-up search ◀── evaluate evidence coverage
                                        │ sufficient
                                        ▼
                              grounded synthesis
                                        │
                                        ▼
                               citation validation

Where it fits

Use web-scout-ai when your application needs the contents of real pages and documents—not only search snippets—and you want the source successes and failures returned as structured data.

It is intentionally a bounded research component. If you only need search links, use a search API directly. If you need an open-ended autonomous research process with human checkpoints, put this package inside a broader agent workflow.

Requirements

• Python 3.13
• An API key for the configured LLM provider
• A Serper or Exa API key for search mode
• Chromium setup for rendered pages, interactive sites, and browser fallbacks

Classification and resource reuse

Semantic classification defaults to TypeSafe Jev and requires TYPESAFE_API_KEY. Set WEB_SCOUT_CLASSIFICATION_BACKEND=gpt to use the existing GPT classifiers instead. Query generation, factual extraction, visual descriptions and synthesis continue to use your configured generative models. Existing follow-up backend selection remains available.

CPU PDF pipelines initialize on the first document and are reused. Browser and HTTP resources are reused on the same event loop; after requests finish, call await web_scout.scraping.close_resources() at application shutdown. See the performance report for measurements, limits, benchmark commands and compatibility details.

Contributing

The main extension point is SearchBackend. New backends should implement the async search() contract and return normalized results (title, url, snippet).

Bug reports and focused pull requests are welcome at github.com/RSO9192/web-scout-ai.

License

MIT