SearchNow

Open-source web search and page extraction for LLMs and AI agents.

One class. Give it a URL and get back the content, the images and the raw HTML. Give it a query and get back search results — from your own SearXNG instance, so nothing leaves your infrastructure and there is no per-query bill.

A self-hosted replacement for Tavily / Brave Search / SerpAPI.

from searchnow import FetchResult

fr = FetchResult()

page = fr.fetch_sync("https://en.wikipedia.org/wiki/Python_(programming_language)")

page.text          # readable markdown, nav and cookie banners stripped
page.html          # the raw HTML source
page.images        # [Image(url=..., alt=..., width=..., height=...), ...]
page.main_image    # the page's own preview image
page.title, page.author, page.description, page.published_date, page.sitename

Install

pip install git+https://github.com/sahiljain/searchnow

Then start SearXNG — one command, and it sets everything up for you:

curl -fsSL https://raw.githubusercontent.com/sahiljain/searchnow/main/setup.sh | bash

That creates a searxng/ folder in the current directory, writes a Docker stack with the JSON API actually enabled, starts it, and verifies it answers. It needs Docker and nothing else.

./setup.sh --port 9090      # a different port
./setup.sh --dir mysearx    # a different folder
./setup.sh --no-start       # write the files, start it yourself

A stock searxng/searxng image will not work: it ships search.formats: [html], so format=json answers 403, and its bot limiter blocks every non-browser client. setup.sh fixes both, plus enables a set of engines that actually respond.

Stop it with cd searxng && docker compose down. More detail, including what that config changes and why, is in docs/searxng.md.

Usage

Fetch a URL

from searchnow import FetchResult

fr = FetchResult()

page = fr.fetch_sync("https://example.com/article")

if page:                        # a Page is truthy when ok
    print(page.title)
    print(page.text)
    for image in page.images:
        print(image.url, image.alt, image.width, image.height)
else:
    print("failed:", page.error)

A failed fetch is a Page with ok=False, not an exception — one dead link must never take down a batch of ten. Pass raise_on_error=True if you want the opposite.

Sites that block scripts

Some sites (Flipkart, and anything behind Cloudflare's challenge, DataDome, PerimeterX…) serve a JavaScript CAPTCHA instead of the page. No plain HTTP client can get past that — not this library, not curl, not requests. SearchNow names the wall instead of leaving you with a bare 403:

page = fr.fetch_sync("https://www.flipkart.com/some-product/p/itm123")

page.ok       # False
page.blocked  # True  <- an anti-bot challenge, not a bug in your code
page.error
# 'HTTP 403 -- blocked by an anti-bot challenge (Google reCAPTCHA). The site
#  only serves this page to a browser that can run its JavaScript challenge...'

blocked means retrying will never work, so branch on it rather than on status_code:

pages = fr.fetch_many_sync(urls)
usable = [p for p in pages if p.ok]
needs_browser = [p.url for p in pages if p.blocked]   # hand these to Playwright

Search

response = fr.search_sync("nvidia nemo framework", max_results=5)

for result in response:
    print(result.score, result.title, result.url)
    print(result.content)               # the engine's snippet

print(response.urls)

Search and read every result — one call

response = fr.search_sync(
    "how does speculative decoding work",
    max_results=5,
    include_content=True,       # downloads all 5 pages concurrently
    max_chars=4000,
)

for result in response:
    print(result.url, len(result.page.text), len(result.page.images))

print(response.to_context())    # numbered, cited markdown for a prompt

Many URLs at once

pages = fr.fetch_many_sync([
    "https://example.com/a",
    "https://example.com/b",
    "https://example.com/c",
])
# Concurrent, input order preserved, failures included as ok=False pages.

Async

Every method has an async twin. Same object, same arguments — drop the _sync:

import asyncio
from searchnow import FetchResult

async def main():
    async with FetchResult() as fr:
        response = await fr.search("retrieval augmented generation", max_results=5)
        pages = await fr.fetch_many(response.urls)
        print(pages[0].text)

asyncio.run(main())

The sync methods run on a private event loop in a background thread, so they also work inside a Jupyter notebook, where asyncio.run would raise "this event loop is already running". You can mix sync and async calls on one instance.

As an LLM tool

from searchnow import FetchResult

fr = FetchResult(cache_ttl=300)      # one instance for the process

async def web_search(query: str, max_results: int = 5) -> str:
    """Search the web and return sourced, readable content."""
    response = await fr.search(query, max_results=max_results, include_content=True)
    return response.to_context(max_chars_per_result=3000)

Keep one FetchResult alive. Building a new one per call throws away the connection pool and the cache, which is the usual reason this feels slow.

Parameters

All optional, all on the constructor, all overridable per call where it makes sense.

fr = FetchResult(
    "http://localhost:8080",     # or $SEARXNG_URL
    max_results=10,
    engines="bing,mojeek",
    cache_ttl=300,
    max_concurrency=16,
)

Per call, search() also takes max_results, engines, categories, language, safesearch, time_range, page, include_content, max_chars, timeout; and fetch() takes timeout, include_html, include_images, max_chars, raise_on_error.

What you get back

Page

Image

.url (always absolute), .alt, .width, .height, .is_main.

Images are found in og:image, twitter:image, JSON-LD, <picture><source srcset> and every <img> — including lazy-loading attributes like data-src, because on a modern page src alone is usually a placeholder spinner. srcset resolves to the largest candidate. Tracking pixels and data: URIs are dropped, and everything is deduplicated.

SearchResponse

Iterable and indexable. .results, .urls, .pages, .answers, .suggestions, .corrections, .infoboxes, .unresponsive_engines, .number_of_results, .elapsed, .cached, .to_dict(), .to_context().

SearchResult

.url, .title, .content (engine snippet), .engine, .engines, .score, .category, .published_date, .thumbnail, .page, .raw.

Accuracy notes

The things that make the difference between "works on example.com" and "works on real sites":

• HTTP/2 is on by default. Wikipedia, Cloudflare-fronted sites and others answer 403 to an HTTP/1.1 request that claims to be a modern browser — because a real Chrome would never speak 1.1 to them. This one setting is the difference between a 403 and a 158 KB article on Wikipedia.
• Two-pass text extraction. trafilatura runs in precision mode first; if that returns suspiciously little, it retries in recall mode before falling back to flattening the document. Short pages stop coming back empty.
• Lazy-loaded images are found, and srcset picks the largest variant.
• Charset detection tries the Content-Type header, then the meta tag, then UTF-8, and never raises on undecodable bytes.
• Retries use exponential backoff with jitter and honour Retry-After, on 429/5xx/timeouts only. A 4xx is never retried.
• Parsing runs off the event loop. lxml over a multi-megabyte document is CPU-bound; left in the loop it would stall every other in-flight fetch.
• Anti-bot walls are named, not swallowed. A 403 from Cloudflare and a 403 from a permission error look identical until you read the body, so SearchNow reads it and tells you which one you hit.

Security

fetch() refuses loopback, private, link-local, multicast and reserved addresses, and re-checks every redirect hop. URLs handed to this method are often produced by a model, so they are attacker-influenced input — this is what stops a prompt-injected http://169.254.169.254/latest/meta-data/, or a public URL that 302s to it, from reading your cloud credentials.

page = fr.fetch_sync("http://169.254.169.254/latest/meta-data/")
page.ok       # False
page.error    # 'URLNotAllowedError: ... resolves to the non-public address ...'

Your SearXNG host is exempt automatically. To reach other internal hosts, opt in:

FetchResult(allowed_hosts=("wiki.internal",))     # one host
FetchResult(allow_private_networks=True)          # everything, use with care

Errors

SearchNowError
├── ConfigurationError          bad constructor arguments
├── TransportError              DNS / TLS / refused / reset
│   └── TimeoutError
├── HTTPStatusError             non-2xx (carries .status_code)
├── SearchBackendError          the instance answered with something unusable
│   └── JSONFormatDisabledError 403 on format=json — the message says how to fix it
├── URLNotAllowedError          blocked by the SSRF guard
└── ContentTypeError            the body is not text

fetch() and fetch_many() convert these into Page.error instead of raising. search() raises them.

Requirements

Python 3.10+, Docker (for SearXNG). Depends on httpx[http2] and trafilatura.

Development

git clone https://github.com/sahiljain/searchnow
cd searchnow
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

make test          # unit tests, fully mocked, no network
make lint          # ruff + mypy --strict
make integration   # needs a live SearXNG

See CONTRIBUTING.md.

License

MIT — see LICENSE.