trawl-search 0.1.0

Self-hosted, Tavily-compatible search API for local development. Point your Tavily SDK at localhost and stop burning credits.

Trawl 🎣

A self-hosted, Tavily-compatible search API for local development.

Point your existing Tavily SDK at localhost and keep building — no credits burned, no monthly limits, no code changes beyond one base-URL argument. When you ship, flip the base URL back and everything behaves the same.

from tavily import TavilyClient

client = TavilyClient(api_key="tvly-anything", api_base_url="http://localhost:7300")
client.search("what is the latest python release?", include_answer=True)
# same request/response shapes, same field names, same error envelope

Why

Agent projects hammer their search tool during development — every test run, every eval loop, every "let me just try this prompt again". Paying per-request (or exhausting a free tier) to debug your own code is silly. Trawl reimplements Tavily's full HTTP API on top of infrastructure you run yourself, so development traffic is free and unlimited, and switching between Trawl and the real Tavily is a one-line change.

What you get

Endpoint	Status	Notes
POST /search	✅	topics, depths, domain filters, time ranges, images, answers, raw content, favicons, usage, auto_parameters, exact_match
POST /extract	✅	markdown/text, query-focused chunking, images, favicons, failed_results
POST /crawl	✅	BFS with regex path/domain selectors, robots.txt, LLM-driven instructions
POST /map	✅	URL discovery

• Wire-compatible: same request fields, same response shapes (follow_up_questions: null and all), same {"detail": {"error": ...}} error envelope both official SDKs parse, HTTP 400/401/403 semantics, Bearer and legacy body api_key auth, keyless mode header.
• No mandatory dependencies on anything paid: search comes from SearXNG (bundled, self-hosted, aggregates Google/Bing/DDG) or DuckDuckGo scraping as a zero-setup fallback.
• Optional local LLM: point TRAWL_LLM_BASE_URL at Ollama / LM Studio / any OpenAI-compatible endpoint to get real generated answers, semantic relevance scores, and instruction-guided crawls. Without one, Trawl degrades gracefully (extractive answers, BM25 scores).
• Optional response cache: TRAWL_CACHE_ENABLED=true makes repeated dev-loop queries instant.

Quick start

Docker (recommended — everything included)

git clone https://github.com/adityaparab/trawl && cd trawl
docker compose up -d
curl -s -X POST http://localhost:7300/search \
  -H "Authorization: Bearer tvly-dev-anything" \
  -H "Content-Type: application/json" \
  -d '{"query": "self-hosted search engines"}' | python -m json.tool

That starts Trawl on :7300 and a pre-configured SearXNG on :8080.

pip / uv (Trawl only)

pip install trawl-search        # or: uv tool install trawl-search
trawl serve

Without a SearXNG instance, set TRAWL_SEARCH_PROVIDER=ddgs for zero-infrastructure DuckDuckGo scraping (fine for light use), or start just SearXNG from this repo's compose file: docker compose up -d searxng.

Check your setup at any time:

trawl check

Pointing your app at Trawl

The official SDKs accept a base-URL override, so no code changes are needed beyond configuration:

Python (tavily-python)

TavilyClient(api_key=os.environ["TAVILY_API_KEY"], api_base_url=os.getenv("TAVILY_BASE_URL", "https://api.tavily.com"))

JavaScript (@tavily/core)

tavily({ apiKey: process.env.TAVILY_API_KEY, apiBaseURL: process.env.TAVILY_BASE_URL })

LangChain (langchain-tavily) — all four tool wrappers accept the override too:

TavilySearch(max_results=5, api_base_url=os.getenv("TAVILY_BASE_URL"))

Set TAVILY_BASE_URL=http://localhost:7300 in development, leave it unset in production. Any non-empty API key works against Trawl by default, so your real TAVILY_API_KEY — or a fake tvly-dev-local — both pass. Runnable versions of all of these are in examples/, and docs/integrations.md covers frameworks without a base-URL option.

Configuration

Copy .env.example to .env next to where you run trawl serve (or the compose file). Everything is optional; defaults work out of the box. Highlights:

Variable	Default	What it does
TRAWL_PORT	7300	API port
TRAWL_SEARCH_PROVIDER	searxng	searxng or ddgs
TRAWL_SEARXNG_URL	http://localhost:8080	your SearXNG instance
TRAWL_PROVIDER_FALLBACK	true	retry via DDGS if SearXNG is down
TRAWL_API_KEYS	(empty)	comma-separated allowlist; empty accepts any non-empty key
TRAWL_LLM_BASE_URL	(empty)	OpenAI-compatible endpoint for answers/instructions
TRAWL_LLM_MODEL	(empty)	chat model name, e.g. llama3.2
TRAWL_EMBEDDING_MODEL	(empty)	if set, semantic scoring via /embeddings
TRAWL_CACHE_ENABLED	false	TTL response cache for dev loops

With Ollama, a full local setup is:

ollama pull llama3.2 && ollama pull nomic-embed-text
cat >> .env <<'EOF'
TRAWL_LLM_BASE_URL=http://localhost:11434/v1
TRAWL_LLM_MODEL=llama3.2
TRAWL_EMBEDDING_MODEL=nomic-embed-text
EOF

Honest differences from Tavily

Trawl is a faithful API clone, not a search-quality clone. Know what you're trading:

• Result quality/order differs. Tavily runs a purpose-built index; Trawl aggregates general-purpose engines and re-ranks locally (BM25, or embeddings when configured). Fine for development; validate ranking-sensitive logic against real Tavily before shipping.
• topic=finance behaves like general. There's no specialized finance index to draw from.
• search_depth: fast/ultra-fast behave like basic. They exist so requests don't fail.
• auto_parameters is a simple heuristic, not Tavily's learned tuning. It's echoed back in the response the same way.
• Scores are comparable within a response, not across services. Don't hard-code thresholds tuned on Tavily scores.
• usage.credits reports what Tavily would charge (approximately, for crawl), so client-side accounting keeps working — no actual metering happens.
• News dates depend on what the upstream engine exposes; undated results are not filtered out by start_date/end_date.

Documentation

docs/api-reference.md	every endpoint, parameter, response shape, and error code
docs/configuration.md	all TRAWL_* variables + Ollama/LM Studio/caching recipes
docs/integrations.md	wiring tavily-python, @tavily/core, LangChain, raw HTTP
docs/compatibility.md	exact parity vs. approximations vs. intentional differences
docs/providers.md	how search backends work / how to add one
docs/deployment.md	compose, systemd, hardening beyond localhost
examples/	runnable snippets for every SDK

Interactive OpenAPI docs are at http://localhost:7300/docs while serving.

Development & contributing

uv venv && uv pip install -e ".[dev]"
.venv/bin/pytest            # offline test suite (mocked providers & pages)
.venv/bin/ruff check src tests
trawl serve --reload

See CONTRIBUTING.md for the parity-first ground rules and project layout, and docs/providers.md for the most-wanted contribution: new search providers (Brave, Kagi, Google CSE, ...). Security reports: SECURITY.md — please read its threat-model section before exposing Trawl beyond localhost. Release history: CHANGELOG.md.

License

MIT. Not affiliated with or endorsed by Tavily — Trawl exists so you can develop against their API shape without spending credits, and hopefully become a happier paying customer in production.