scoutlet 0.2.0

Lightweight local search aggregation tool for AI agents, compatible with SearXNG 200+ engines

scoutlet

English | 简体中文

License: AGPL-3.0 | Notice

A minimalist local search aggregation tool for AI agents — no API keys, no heavy dependencies, compatible with SearXNG's 200+ search engine ecosystem, delivering powerful search capabilities to local agents.

Based on SearXNG's engine system and result aggregation algorithm, retaining only the core engine loading, concurrent requests, scoring, deduplication, merging, and sorting logic.

Features

• Only 3 core dependencies: httpx, lxml, babel
• Reuses SearXNG's result aggregation algorithm (weighted scoring, hash dedup, merge, group sorting)
• Compatible with SearXNG engine code patterns — copy and change imports to use
• Python API and CLI interfaces
• 108 built-in engines (general, news, images, videos, code, music, files, science, movies, social media, and more)
• Optional TLS fingerprint backend (primp) for browser-grade TLS impersonation
• Engine health monitoring + AI auto-heal pipeline (snapshots → LLM repair → PR)

Why scoutlet

Comparison

There are several approaches for AI agents to acquire search capabilities:

	scoutlet	Search API Services	SearXNG	Search + LLM Frameworks
Examples	-	Tavily, Exa, SerpAPI	SearXNG	OpenDeepSearch, MindSearch
Runtime	Python library, embed directly	Call paid APIs	Requires web service deployment	Framework-level, tied to LLM
Dependencies	None	Requires API Key	Flask + Docker	Specific LLM + search API
Core deps	3 (httpx/lxml/babel)	SDK or HTTP	Dozens	Varies
Anti-bot	TLS fingerprint adapter (optional)	Handled by provider	None	Depends on external API
Engine ecosystem	Compatible with SearXNG engines	Fixed engines	200+ engines	Fixed engines
Result quality	Multi-engine aggregated scoring	Single engine	Multi-engine aggregated scoring	Varies
Cost	Free, runs locally	Paid / rate-limited	Free but requires server	Free + LLM costs

scoutlet's positioning: SearXNG's search capabilities, delivered as a pure Python library that embeds directly into agents or via CLI/MCP service — zero external services, zero API keys, zero deployment.

Highlights

1. Local, service-free search aggregation library

Shares SearXNG's engine ecosystem and aggregation algorithm (weighted scoring, hash dedup, merge/sort), but transforms from a web service into an embeddable Python library. No deployment needed — pip install and go.

2. Ultra-lightweight, 3 core dependencies

Only httpx, lxml, babel. Compared to SearXNG's dozens of server-side dependencies, scoutlet runs truly zero-config in any Python environment.

3. SearXNG engine compatibility

Directly reuse SearXNG's massive engine ecosystem (200+). Copy engine files from SearXNG and change imports — no other solution offers this level of engine reuse.

Built-in Engines (108)

Engines grouped by primary category. The "also in" column lists additional categories the engine belongs to. Engines with categories = [] (emojipedia, wikipedia) are listed at the end — they are not picked up by default searches and must be invoked with -e <name>.

general (12) — default category

Engine	also in	Description
google		Google Web Search
bing	web	Bing Web Search
brave	web	Brave Search
duckduckgo		DuckDuckGo HTML (no-JS)
yahoo	web	Yahoo Search
qwant	web	Qwant Search
baidu		Baidu Search
sogou		Sogou Search
quark		Quark/Shenma Search
seznam		Seznam Search
marginalia		Marginalia Search
mwmbl		Mwmbl Search

images (24)

Engine	also in	Description
google_images	web	Google Images
bing_images	web	Bing Images
duckduckgo_extra	videos, news	DuckDuckGo Extra (set ddg_category=images)
unsplash		Unsplash Photos (requires API key)
imgur		Imgur Images
wallhaven		Wallhaven Wallpapers (requires API key)
artic		Art Institute of Chicago
artstation		ArtStation Artworks
deviantart		DeviantArt
findthatmeme		FindThatMeme
flickr		Flickr (requires API key)
flickr_noapi		Flickr (no API key)
ipernity		Ipernity
loc		Library of Congress Photos
openclipart		OpenClipArt
openverse		Openverse CC Media
pexels		Pexels Photos
pinterest		Pinterest
pixabay		Pixabay Media
pixiv		Pixiv Illustrations
public_domain_image_archive		Public Domain Image Archive
sogou_images		Sogou Images
www1x		1x Photography
frinkiac		Frinkiac Simpsons Screenshots

videos (24)

Engine	also in	Description
google_videos	web	Google Videos
bing_videos	web	Bing Videos
youtube_noapi	music	YouTube (no API key)
youtube_api	music	YouTube Data API v3 (requires API key)
vimeo		Vimeo Videos
invidious		Invidious (YouTube front-end)
piped		Piped (YouTube front-end)
bilibili		Bilibili Videos
360search_videos		360Search Videos
acfun		Acfun Videos
bitchute		Bitchute Videos
ccc_media		media.ccc.de
dailymotion		Dailymotion Videos
digbt	music, files	DigBT Torrents
ina		INA (French)
iqiyi		iQiyi Videos
mediathekviewweb		MediathekViewWeb (German)
niconico		Niconico Videos
odysee		Odysee Videos
peertube		Peertube Federated Videos
rumble		Rumble Videos
sepiasearch		SepiaSearch Federated Videos
sogou_videos		Sogou Videos
tubearchivist		Tube Archivist (self-hosted, requires base_url+token)

music (8)

Engine	also in	Description
deezer		Deezer Music
genius	lyrics	Genius Lyrics
bandcamp		Bandcamp Music
mixcloud		Mixcloud
radio_browser	radio	Radio Browser Stations
soundcloud		SoundCloud
spotify		Spotify (requires client credentials)
yandex_music		Yandex Music

news (7)

Engine	also in	Description
google_news		Google News
bing_news		Bing News
yahoo_news		Yahoo News
sogou_wechat		Sogou WeChat Articles
ansa		Ansa (Italian)
il_post		Il Post (Italian)
reuters		Reuters

it (10)

Engine	also in	Description
github	repos	GitHub Repos Search
gitlab	repos	GitLab Repos Search (requires base_url)
gitea	repos	Gitea/Forgejo Repos Search
sourcehut	repos	SourceHut Repos Search
npm	packages	NPM Packages
docker_hub	packages	Docker Hub Images
crates	packages	Rust crates
pypi	packages	PyPI Packages
stackexchange	q&a	Stack Exchange (StackOverflow)
hackernews	news	Hacker News

science (8)

Engine	also in	Description
arxiv	scientific publications	arXiv Preprints
crossref	scientific publications	Crossref Scholarly Metadata
openalex	scientific publications	OpenAlex Works
semantic_scholar	scientific publications	Semantic Scholar Papers
pubmed	scientific publications	PubMed Biomedical Literature
pdbe		PDBe Protein Structures
astrophysics_data_system	scientific publications	NASA ADS (requires API key)
scanr_structures		ScanR French Research Structures

social media (6)

Engine	also in	Description
reddit		Reddit Search
9gag		9GAG
lemmy		Lemmy (Communities/Users/Posts/Comments)
mastodon		Mastodon (accounts/hashtags)
mrs		Matrix Rooms Search (requires base_url)
tootfinder		Tootfinder (Mastodon posts)

movies (4)

Engine	also in	Description
imdb		IMDB
moviepilot		Moviepilot (German)
rottentomatoes		Rotten Tomatoes
senscritique		SensCritique (French)

files (2)

Engine	also in	Description
1337x		1337x Torrents
nyaa		Nyaa Anime Torrents

code (1)

Engine	also in	Description
github_code	it	GitHub Code Search

No category (2) — must invoke with -e <name>

These engines have categories = [], matching SearXNG. Default searches skip them.

Engine	Description
emojipedia	Emojipedia Emoji Reference
wikipedia	Wikipedia Summary

Installation

pip install -e .

# For TLS fingerprint backend (optional)
pip install -e ".[fingerprint]"

Requires Python >= 3.10.

Usage

Python API

from scoutlet import search_sync, search

# Synchronous search (recommended for scripts)
response = search_sync("python tutorial", engines=["google", "bing"])

# Results live under .results
for r in response.results:
    print(f"[{','.join(r.engines)}] {r.title}")
    print(f"  {r.url}")
    print(f"  {r.content[:100]}")
    print(f"  score: {r.score:.2f}")

# Diagnostics: which engines failed, which were skipped (cooldown etc.)
for e in response.failed:
    print(f"  engine {e.name}: {e.status.value} - {e.error}")
for s in response.skipped:
    print(f"  skipped {s.name}: {s.reason}")

# Async search (for use in async programs)
response = await search("python tutorial", engines=["google", "bing"])

# Search by category
response = search_sync("AI", categories=["general", "news"])

# Specify language and time range
response = search_sync("latest news", language="zh", time_range="day")

CLI

# Search
scoutlet "python tutorial" -e google,bing

# JSON output
scoutlet "python tutorial" -e google,bing -f json

# Specify language, time range
scoutlet "latest news" -l zh -t day -e baidu,sogou

# List available engines
scoutlet --list-engines

# Group engines by category
scoutlet --list-engines --by-category

Proxy

HTTP Proxy

Support HTTP/SOCKS5 proxy via the proxy parameter for all engines:

# HTTP proxy
results = search("test", engines=["google"], proxy="http://127.0.0.1:7890")

# SOCKS5 proxy (requires: pip install httpx[socks])
results = search("test", engines=["google"], proxy="socks5://127.0.0.1:1080")

scoutlet "test" -e google --proxy http://127.0.0.1:7890
# SOCKS5 requires: pip install httpx[socks]
scoutlet "test" -e google --proxy socks5://127.0.0.1:1080

Google Engine Notes

Google search uses a mobile GSA User-Agent (Android Chrome) to bypass JS-only pages and return traditional HTML. As Google's anti-bot strategies evolve, you may need to:

# Update GSA User-Agent list
python scripts/update_gsa_useragents.py

If the GSA UA is also blocked, switch to the TLS fingerprint backend (see below) or pick an alternative engine from the default set.

TLS Fingerprint HTTP Backend (Optional)

Some engines detect bot traffic at the TLS layer (JA3/JA4 fingerprint, HTTP/2 settings). scoutlet ships an optional primp-based adapter that impersonates a random real browser (Chrome/Firefox/Safari) — cipher suites, ALPN, HTTP/2 frames, and header order all match a real browser, so the request is indistinguishable at the TLS layer.

pip install -e ".[fingerprint]"   # installs primp>=1.2.3

Per-engine (recommended — only fingerprint the engines that need it):

load_engines(engine_configs={
    "duckduckgo": {"http_client": "fingerprint"},
    "google": {"http_client": "fingerprint"},
})

results = search_sync("test", engines=["google"])

Global (applies to every engine):

from scoutlet.network import set_adapter_backend

set_adapter_backend("fingerprint")

Leave http_client empty / unset to keep the default httpx backend. Custom adapters can be registered via scoutlet.client_adapter.register_adapter(name, cls).

Engine Health & Auto-Heal

scoutlet ships a CI pipeline (.github/workflows/engine-health.yml) that runs every 6 hours, performs an end-to-end probe of all engines, classifies failures, and triggers an LLM-driven auto-heal flow that opens a fix PR when a parser breaks.

Pipeline stages:

1. Health check (scripts/health_check.py --all) — runs a real search per engine, classifies status as healthy / empty / anti_bot / http_error / parser_error / timeout, dumps JSON report + saves failed HTML snapshots.
2. Snapshot manager (scripts/snapshot.py) — stores baseline_* and failed_* HTML side-by-side, with a minimizer that strips <script>/<style> and large blobs for cheaper LLM input.
3. AI repair agent (scripts/auto_heal.py) — for each parse failure, reads the current engine source, the failed HTML, and the baseline HTML, prompts an OpenAI-compatible LLM to rewrite the response() parser, then verifies the result with: forbidden-pattern scan, ast.parse syntax check, fixture replay, and a live re-probe.
4. PR creation — if any engine was successfully repaired, the workflow commits to an auto-fix/<timestamp> branch and opens a PR labeled auto-fix for human review.

Configuration (CI secrets):

Secret	Purpose
OPENAI_API_KEY	LLM provider key
OPENAI_API_BASE	OpenAI-compatible base URL
OPENAI_MODEL	Model name (default gpt-4o)

Run locally:

# Probe all engines, save snapshots
uv run python scripts/health_check.py --all --output health-report.json --snapshots-dir snapshots

# Attempt repairs from the latest report
uv run python scripts/auto_heal.py --report health-report.json --snapshots-dir snapshots --dry-run

See design doc for the full architecture.

Tests

593 offline tests across three suites:

uv run pytest tests/unit/        # core logic: result types, aggregation, engine_loader, network, response_classifier, client_adapter, CLI
uv run pytest tests/engines/     # P0/P1 engine parser fixtures (saved HTML/JSON, no network)
uv run pytest tests/             # all offline tests

Live smoke tests against real engines are gated by pytest.mark.live and require SCOUTLET_LIVE=1:

SCOUTLET_LIVE=1 uv run pytest tests/live/ -m live

CI (.github/workflows/engine-health.yml) runs compileall + offline tests + the health/auto-heal pipeline on every 6-hour schedule.

TODO

• Sync more SearXNG engines, complete migration and testing of all 200+ engines, periodically sync upstream SearXNG engine updates
• Continuously follow SearXNG's anti-bot strategies (UA updates, request parameter adjustments, new engine adaptations, etc.)
• Engine Auto-Heal — Automated engine health monitoring and AI-powered self-repair system
  • Health Monitor: scheduled end-to-end testing of all engines (scripts/health_check.py + CI every 6h)
  • HTML Snapshot: save baseline and failed HTML responses for comparison (scripts/snapshot.py)
  • AI Repair Agent: when parsing fails, call LLM to analyze HTML structure changes and rewrite engine parsing code (scripts/auto_heal.py)
  • Auto-Tester: full sandbox validation of AI-generated code (currently CI does compileall + fixture replay + live re-probe; isolated sandbox still TODO)
  • Auto-commit: verified fixes committed via PR for review (CI auto-heal job)
  • See design doc for details

License

scoutlet is licensed under the GNU Affero General Public License v3. Engine code under src/scoutlet/engines/ is derived from SearXNG (also AGPL-3.0). See NOTICE for third-party attributions.