Unique Web Search
A powerful, configurable web search tool for retrieving and processing the latest information from the internet. This package provides intelligent search capabilities with support for multiple search engines, web crawlers, and content processing strategies.
Architecture
The following diagram illustrates the complete architecture and workflow of the unique_web_search package:
Key Features
-
Dual Execution Modes:
- V1 (Traditional): Query refinement with single or multiple search strategies
- V2 (Step-based Planning): Advanced research planning with parallel execution
-
Multiple Search Engines:
- Google Search
- Bing Search
- Brave Search
- Jina Search
- Tavily Search
- Firecrawl Search
- VertexAI (Gemini with Grounding)
- Custom API (integrate any compatible web search API)
-
Multiple Web Crawlers:
- Basic HTTP Crawler
- Crawl4AI
- Jina Reader
- Tavily Crawler
- Firecrawl Crawler
-
Intelligent Content Processing:
- LLM-based summarization
- Token-based truncation
- Relevancy scoring and sorting
- Content chunking and optimization
-
Query Refinement:
- BASIC Mode: Single optimized search query
- ADVANCED Mode: Multiple targeted search queries for complex research
-
Performance Optimized:
- Parallel execution of search and crawl operations
- Token limit management
- Configurable timeouts and error handling
Detailed Subsystem Docs
For deeper dives into each subsystem, see the dedicated READMEs:
- Search Engines — full catalogue of supported engines, configuration, and usage examples.
- Crawlers — comparison of crawling strategies (Basic, Crawl4AI, Tavily, Firecrawl, Jina) with setup guides.
- Executors — orchestration layer (V1 & V2) covering query refinement, planning, logging, and best practices.
Configuration
The tool uses environment variables and configuration files to manage API keys and settings. Key configuration areas include:
- Search engine selection and API keys
- Crawler selection and configuration
- Content processing strategies (SUMMARIZE, TRUNCATE, NONE)
- Token limits and relevancy thresholds
- Proxy configuration
- Debug and monitoring options
Dependency management (uv.lock + min/latest testing)
This package is a library and uses uv for dependency management.
We run tests additionally with minimal dependencies to ensure that the listed ranges are valid. NOTE: We use lowest-direct, not lowest. Lowest attempts to use the lowest possible dependency versions tarnsitively causing issues if a dependency has incorrect metadata. Example:
- google-cloud-aiplatform says it works with shapely<3.0.0.
- The lowest resolver assumes 1.0 which needs python 2 -> breaks Therefore we use lowest-direct which only sets our direct dependencies to lowest. However, this only correctly verifies our min dependencies if our code correctly lists all the required dependencies and never imports a transitive dependency. We therefore use deptry to ansure we don't use transitive dependencies and that we have no unused dependencies.
Test locally
- Latest deps and deptry:
cd tool_packages/unique_web_search
uv sync
uv run pytest
uv run deptry
- Min deps:
cd tool_packages/unique_web_search
uv venv
# Install runtime deps at minimum versions
uv pip install -e . --resolution=lowest-direct
# Install dev deps from [dependency-groups] (we only care about runtime dep minimums)
uv export --only-group dev --no-hashes | uv pip install -r -
# Use --no-sync to prevent uv from "fixing" the versions
uv run --no-sync pytest
Workflow
- Input: User query or structured search plan
- Configuration: Load settings and initialize services
- Execution:
- V1: Query refinement → Search → Crawl → Process
- V2: Execute planned steps in parallel → Process
- Content Processing: Clean, summarize/truncate, and chunk content
- Optimization: Reduce to token limits and sort by relevance
- Output: Return structured content chunks optimized for LLM consumption
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file unique_web_search-2026.28.1.tar.gz.
File metadata
- Download URL: unique_web_search-2026.28.1.tar.gz
- Upload date:
- Size: 113.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.10.11 {"installer":{"name":"uv","version":"0.10.11","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
6754f4777c7d162b0318aaf1739da86b28440f5a2f583c27076b47a743f52748
|
|
| MD5 |
1437779acb7c6a70c064b1afc28fcda1
|
|
| BLAKE2b-256 |
71561751afb4c5d14a8850951fcdd1597dd964ca479e7d81112e3cb76babc096
|
File details
Details for the file unique_web_search-2026.28.1-py3-none-any.whl.
File metadata
- Download URL: unique_web_search-2026.28.1-py3-none-any.whl
- Upload date:
- Size: 171.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.10.11 {"installer":{"name":"uv","version":"0.10.11","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8d55956690af5d16f7b860f7a1a013a92ed992c2cad4029ffa457b9ed83838cc
|
|
| MD5 |
f7fb07f763308f28fddc09a52a9595a6
|
|
| BLAKE2b-256 |
17658e81b2ce5a0e5267d6a389c61f3496a307a1cc9d03993a306f3e36fb26e2
|
