SearchNow

Open-source web search and page extraction for LLMs and AI agents.

One class. Give it a URL and get back the content, the images and the raw HTML. Give it a query and get back search results — from your own SearXNG instance, so nothing leaves your infrastructure and there is no per-query bill.

A self-hosted replacement for Tavily / Brave Search / SerpAPI.

from searchnow import FetchResult

fr = FetchResult()

page = fr.fetch_sync("https://en.wikipedia.org/wiki/Python_(programming_language)")

page.text          # readable markdown, nav and cookie banners stripped
page.html          # the raw HTML source
page.images        # [Image(url=..., alt=..., width=..., height=...), ...]
page.main_image    # the page's own preview image
page.title, page.author, page.description, page.published_date, page.sitename

---

Install

pip install git+https://github.com/sahiljain/searchnow

That's the whole setup. You need Docker installed and running; SearchNow handles the rest. The first time you search(), if no SearXNG is answering, it starts one for you:

from searchnow import FetchResult

fr = FetchResult()
fr.search_sync("nvidia nemo framework")
# WARNING no SearXNG instance answering on port 8080 -- setting one up in
#         ~/.searchnow/searxng (the first run pulls a ~500 MB Docker image...)
# WARNING SearXNG is up. Stop it with: cd ~/.searchnow/searxng && docker compose down

Roughly 2-4 minutes the first time (Docker image pull), a couple of seconds after that. If you delete the folder or stop the containers, the next search() rebuilds it.

How it decides

Something already answering on base_url	does nothing
Nothing there, base_url is local	runs the setup
Nothing there, base_url is remote	raises — it will not "fix" someone else's server by starting a local container
Docker missing or not running	raises, and says exactly what to install or start
You called fetch(), not search()	does nothing — fetching a URL needs no SearXNG

The script (setup.sh) ships inside the package, so nothing is downloaded at runtime. You can also run it yourself:

script=$(python -c "import searchnow; print(searchnow.bootstrap.script_path())")
bash "$script" --dir searxng --port 8080
#   --port 9090     a different port
#   --dir mysearx   a different folder (its own Compose project, no collisions)
#   --force         overwrite an existing config
#   --no-start      write the files, start it yourself

Turning it off

fr = FetchResult("https://searx.mycompany.internal", auto_setup=False)

or export SEARCHNOW_AUTO_SETUP=0. (An env var can only turn it off — an environment should not be able to opt you into starting containers.) You can also control when the slow first setup happens, rather than having it land mid-request:

fr = FetchResult()
fr.ensure_searxng_sync()      # at startup; await fr.ensure_searxng() in async code

A stock searxng/searxng image will not work: it ships search.formats: [html], so format=json answers 403, and its bot limiter blocks every non-browser client. setup.sh fixes both, plus enables a set of engines that actually respond. Details in docs/searxng.md.

---

Usage

Fetch a URL

from searchnow import FetchResult

fr = FetchResult()

page = fr.fetch_sync("https://example.com/article")

if page:                        # a Page is truthy when ok
    print(page.title)
    print(page.text)
    for image in page.images:
        print(image.url, image.alt, image.width, image.height)
else:
    print("failed:", page.error)

A failed fetch is a Page with ok=False, not an exception — one dead link must never take down a batch of ten. Pass raise_on_error=True if you want the opposite.

fetch() takes a URL, search() takes a query. The 0.0.x names still work and map the same way: fetch_url → fetch, fetch_data → search. Passing a query to fetch() tells you so rather than failing at DNS:

fr.fetch_sync("nemo").error
# "this is not a URL. fetch() downloads one page ... To look a topic up,
#  use search() instead -- e.g. fr.search_sync('nemo')"

Sites that block scripts

Some sites (Flipkart, and anything behind Cloudflare's challenge, DataDome, PerimeterX…) serve a JavaScript CAPTCHA instead of the page. No plain HTTP client can get past that — not this library, not curl, not requests. SearchNow names the wall instead of leaving you with a bare 403:

page = fr.fetch_sync("https://www.flipkart.com/some-product/p/itm123")

page.ok       # False
page.blocked  # True  <- an anti-bot challenge, not a bug in your code
page.error
# 'HTTP 403 -- blocked by an anti-bot challenge (Google reCAPTCHA). The site
#  only serves this page to a browser that can run its JavaScript challenge...'

blocked means retrying will never work, so branch on it rather than on status_code:

pages = fr.fetch_many_sync(urls)
usable = [p for p in pages if p.ok]
needs_browser = [p.url for p in pages if p.blocked]   # hand these to Playwright

Search

response = fr.search_sync("nvidia nemo framework", max_results=5)

for result in response:
    print(result.score, result.title, result.url)
    print(result.content)               # the engine's snippet

print(response.urls)

Search and read every result — one call

response = fr.search_sync(
    "how does speculative decoding work",
    max_results=5,
    include_content=True,       # downloads all 5 pages concurrently
    max_chars=4000,
)

for result in response:
    print(result.url, len(result.page.text), len(result.page.images))

print(response.to_context())    # numbered, cited markdown for a prompt

Many URLs at once

pages = fr.fetch_many_sync([
    "https://example.com/a",
    "https://example.com/b",
    "https://example.com/c",
])
# Concurrent, input order preserved, failures included as ok=False pages.

Async

Every method has an async twin. Same object, same arguments — drop the _sync:

sync	async
fr.search_sync(q)	await fr.search(q)
fr.fetch_sync(url)	await fr.fetch(url)
fr.fetch_many_sync(urls)	await fr.fetch_many(urls)

import asyncio
from searchnow import FetchResult

async def main():
    async with FetchResult() as fr:
        response = await fr.search("retrieval augmented generation", max_results=5)
        pages = await fr.fetch_many(response.urls)
        print(pages[0].text)

asyncio.run(main())

The sync methods run on a private event loop in a background thread, so they also work inside a Jupyter notebook, where asyncio.run would raise "this event loop is already running". You can mix sync and async calls on one instance.

As an LLM tool

from searchnow import FetchResult

fr = FetchResult(cache_ttl=300)      # one instance for the process

async def web_search(query: str, max_results: int = 5) -> str:
    """Search the web and return sourced, readable content."""
    response = await fr.search(query, max_results=max_results, include_content=True)
    return response.to_context(max_chars_per_result=3000)

Keep one FetchResult alive. Building a new one per call throws away the connection pool and the cache, which is the usual reason this feels slow.

---

Parameters

All optional, all on the constructor, all overridable per call where it makes sense.

fr = FetchResult(
    "http://localhost:8080",     # or $SEARXNG_URL
    max_results=10,
    engines="bing,mojeek",
    cache_ttl=300,
    max_concurrency=16,
)

Parameter	Default	What it does
base_url	$SEARXNG_URL or http://localhost:8080	Your SearXNG instance
max_results	5	Default result count
engines	(instance default)	"bing,mojeek" or ["bing", "mojeek"]
categories	"general"	general, news, science, it, images, …
language	"en"	Result language
safesearch	0	0 off, 1 moderate, 2 strict
time_range	None	day / week / month / year
search_timeout	30.0	Seconds
fetch_timeout	20.0	Seconds
max_retries	2	Retries after the first attempt
backoff_factor	0.5	Base of the exponential backoff
max_bytes	2_000_000	Body cap per page; oversized pages truncate
max_concurrency	8	Parallel fetches
max_redirects	5	Hops per fetch
include_html	True	Keep raw HTML on the Page
include_images	True	Collect images
max_images	50	Cap per page
user_agent	Chrome UA	Sent on every request
headers	None	Extra headers, merged in
proxy	None	e.g. http://127.0.0.1:8118
verify_ssl	True	TLS verification
http2	True	Leave it on — see below
allow_private_networks	False	See Security
allowed_hosts	()	Hosts exempt from the SSRF check
cache_ttl	0 (off)	Seconds to reuse an identical search or fetch
cache_size	256	Cache entries
auto_setup	True	Start SearXNG on first search() if none is up
setup_dir	~/.searchnow/searxng	Where auto-setup puts the stack

Per call, search() also takes max_results, engines, categories, language, safesearch, time_range, page, include_content, max_chars, timeout; and fetch() takes timeout, include_html, include_images, max_chars, raise_on_error.

---

What you get back

Page

.ok	Did it work. Page is truthy when it did
.text	Readable markdown — nav, cookie banners and footers gone
.html	Raw HTML source
.images	list[Image], the page's own preview image first
.main_image	That preview image's URL, or None
.title .description .author .published_date .sitename .language	Metadata
.url .final_url .status_code .content_type	Where it came from
.error .elapsed .truncated .fetched_at	Diagnostics
.to_dict()	JSON-ready

Image

.url (always absolute), .alt, .width, .height, .is_main.

Images are found in og:image, twitter:image, JSON-LD, <picture><source srcset> and every <img> — including lazy-loading attributes like data-src, because on a modern page src alone is usually a placeholder spinner. srcset resolves to the largest candidate. Tracking pixels and data: URIs are dropped, and everything is deduplicated.

SearchResponse

Iterable and indexable. .results, .urls, .pages, .answers, .suggestions, .corrections, .infoboxes, .unresponsive_engines, .number_of_results, .elapsed, .cached, .to_dict(), .to_context().

SearchResult

.url, .title, .content (engine snippet), .engine, .engines, .score, .category, .published_date, .thumbnail, .page, .raw.

---

Accuracy notes

The things that make the difference between "works on example.com" and "works on real sites":

• HTTP/2 is on by default. Wikipedia, Cloudflare-fronted sites and others answer 403 to an HTTP/1.1 request that claims to be a modern browser — because a real Chrome would never speak 1.1 to them. This one setting is the difference between a 403 and a 158 KB article on Wikipedia.
• Two-pass text extraction. trafilatura runs in precision mode first; if that returns suspiciously little, it retries in recall mode before falling back to flattening the document. Short pages stop coming back empty.
• Lazy-loaded images are found, and srcset picks the largest variant.
• Charset detection tries the Content-Type header, then the meta tag, then UTF-8, and never raises on undecodable bytes.
• Retries use exponential backoff with jitter and honour Retry-After, on 429/5xx/timeouts only. A 4xx is never retried.
• Parsing runs off the event loop. lxml over a multi-megabyte document is CPU-bound; left in the loop it would stall every other in-flight fetch.
• Anti-bot walls are named, not swallowed. A 403 from Cloudflare and a 403 from a permission error look identical until you read the body, so SearchNow reads it and tells you which one you hit.

---

Security

fetch() refuses loopback, private, link-local, multicast and reserved addresses, and re-checks every redirect hop. URLs handed to this method are often produced by a model, so they are attacker-influenced input — this is what stops a prompt-injected http://169.254.169.254/latest/meta-data/, or a public URL that 302s to it, from reading your cloud credentials.

page = fr.fetch_sync("http://169.254.169.254/latest/meta-data/")
page.ok       # False
page.error    # 'URLNotAllowedError: ... resolves to the non-public address ...'

Your SearXNG host is exempt automatically. To reach other internal hosts, opt in:

FetchResult(allowed_hosts=("wiki.internal",))     # one host
FetchResult(allow_private_networks=True)          # everything, use with care

---

Errors

SearchNowError
├── ConfigurationError          bad constructor arguments
├── TransportError              DNS / TLS / refused / reset
│   └── TimeoutError
├── HTTPStatusError             non-2xx (carries .status_code)
├── SearchBackendError          the instance answered with something unusable
│   └── JSONFormatDisabledError 403 on format=json — the message says how to fix it
├── URLNotAllowedError          blocked by the SSRF guard
└── ContentTypeError            the body is not text

fetch() and fetch_many() convert these into Page.error instead of raising. search() raises them.

---

Requirements

Python 3.10+, Docker (for SearXNG). Depends on httpx[http2] and trafilatura.

---

Development

git clone https://github.com/sahiljain/searchnow
cd searchnow
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

make test          # unit tests, fully mocked, no network
make lint          # ruff + mypy --strict
make integration   # needs a live SearXNG

See CONTRIBUTING.md.

---

License

MIT — see LICENSE.