evidence-fetcher 0.2.0

Interactive evidence-search and relevance-feedback experiment scaffold.

evidence-fetcher

evidence-fetcher is an interactive evidence-search and relevance-feedback experiment.

The current vertical slice fetches candidate web evidence from Brave Search, normalizes the provider response, removes duplicate URLs, applies a lightweight local relevance ranker, and exposes the results through a Python API and a small command-line interface.

Installation

Install the core package:

pip install evidence-fetcher

Configuration

Create a Brave Search API key and expose it as BRAVE_SEARCH_API_KEY:

export BRAVE_SEARCH_API_KEY="..."

Python API

from evidence_fetcher import fetch_evidence

results = fetch_evidence("retrieval augmented generation evaluation")

for evidence in results:
    print(evidence.title)
    print(evidence.url)
    print(evidence.snippet)

fetch_evidence() returns typed Evidence objects with normalized title, url, snippet, source, and rank fields.

CLI

Readable text output:

evidence-fetcher "retrieval augmented generation evaluation"

JSON output:

evidence-fetcher "retrieval augmented generation evaluation" --format json

Use --limit to control the maximum number of results.

Interactive relevance feedback:

evidence-fetcher "retrieval augmented generation evaluation" --interactive

Interactive mode performs the same initial search, then lets you mark displayed results, request a local rerank, or explicitly expand retrieval with a refined query. Ordinary text and JSON commands are unchanged and remain non-interactive. Interactive mode is text-only; combining --interactive with --format json is rejected.

Supported interactive commands:

more 1 3
less 4
irrelevant 5
unsure 2
show
expand
help
done

Result numbers are presentation-only positions in the currently displayed list. Feedback is stored against each evidence URL, and after show the same number may refer to a different result.

show reranks only the current candidate set using accumulated feedback. expand builds a deterministic lexical query from the original query and feedback-marked evidence, performs one additional Brave retrieval, merges any new URLs into the session, preserves accumulated feedback, and reranks the expanded candidate set. Expansion is always user-triggered; feedback commands do not automatically run new searches.

Example session:

$ evidence-fetcher "retrieval augmented generation evaluation" --interactive
Interactive relevance feedback. Type 'help' for commands.
1. Result title
   https://example.com/page
   Short snippet...
> more 1
recorded more: 1
> irrelevant 3
recorded irrelevant: 3
> show
Updated ranking from accumulated feedback:
1. Result title
   https://example.com/page
   Short snippet...
> expand
Searching with refined query:
retrieval augmented generation evaluation benchmark faithfulness
Added 4 new results.
Updated ranking:
1. Result title
   https://example.com/page
   Short snippet...
> done

Architecture

The public API is intentionally small: fetch_evidence() accepts a query and returns normalized, deduplicated Evidence results. Provider-specific HTTP requests, credential handling, and response parsing live behind an internal Brave Search provider. The provider depends on a small HTTP client protocol, so tests and future integrations can inject a mock client instead of using the network.

Candidate retrieval and local relevance ranking are separate steps. Brave Search retrieves candidate evidence from the web; a replaceable local ranker then orders those candidates against the original query. The default ranker is a small standard-library TF-IDF implementation over evidence titles and snippets. This is the first foundation for Interactive Information Retrieval: the code now has a boundary where a workflow can collect user relevance feedback and rerank results.

Feedback-aware reranking is available through the Python/domain layer. A SearchSession retains the original query, candidate evidence, current ranked results, and accumulated RelevanceFeedback entries keyed by evidence URL. The TF-IDF ranker applies a lightweight Rocchio-style query update: it keeps the original query vector, moves it toward evidence marked MORE or RELEVANT, moves it away from evidence marked LESS or IRRELEVANT, and records UNSURE without changing ranking.

Feedback-informed retrieval expansion is separate from local reranking. The default query refiner tokenizes the original query and feedback-marked evidence, removes small stop words, scores candidate expansion terms lexically with TF-IDF, avoids duplicating original query terms, suppresses terms associated mainly with irrelevant evidence, and appends up to four useful terms. The refined query is printed before each expand retrieval. Expansion does not persist a session across invocations, run in the background, use an LLM, or send feedback anywhere except the explicit refined Brave query.

Package-specific exceptions distinguish configuration errors, request failures, and malformed provider responses.

Only Brave Search is implemented in this slice. No Streamlit UI, persistence, or additional search providers are included yet.

Development

This project uses uv for environment and dependency management.

uv sync
uv run pytest
uv run ruff check .
uv run mypy src
uv build
uv run sphinx-build -b html docs/source docs/built

Common shortcuts are available through make:

make check
make docs
make clean

Documentation

Documentation is built with Sphinx and Furo from docs/source/ into docs/built/. Read the Docs should build the documentation through its normal repository integration.

Releases

Releases are managed with Python Semantic Release and Conventional Commits. The GitHub Actions release workflow is intended to create the release commit, tag, GitHub release, wheel, sdist, and PyPI publication through trusted publishing.