evidence-fetcher 0.3.0

Interactive evidence-search and relevance-feedback experiment scaffold.

evidence-fetcher

evidence-fetcher is an interactive evidence-search and relevance-feedback experiment.

The current vertical slice fetches candidate web evidence from Brave Search, normalizes the provider response, removes duplicate URLs, applies a lightweight local relevance ranker, and exposes the results through a Python API and a small command-line interface.

The feedback-aware reranking is based on a Rocchio-style relevance feedback algorithm over TF-IDF vectors.

Because Interactive Information Retrieval can amplify the feedback you provide, it should be used with care. The goal is to condense useful evidence from search results, not to cherry-pick sources that only support a preferred conclusion. Reviewers should keep contrary, ambiguous, and low-confidence evidence visible when it matters to the question.

Installation

Install the core package:

pip install evidence-fetcher

Configuration

Create a Brave Search API key and expose it as BRAVE_SEARCH_API_KEY:

export BRAVE_SEARCH_API_KEY="..."

Python API

from evidence_fetcher import fetch_evidence

results = fetch_evidence("retrieval augmented generation evaluation")

for evidence in results:
    print(evidence.title)
    print(evidence.url)
    print(evidence.snippet)

fetch_evidence() returns typed Evidence objects with normalized title, url, snippet, source, and rank fields.

CLI

Readable text output:

evidence-fetcher "retrieval augmented generation evaluation"

JSON output:

evidence-fetcher "retrieval augmented generation evaluation" --format json

Use --limit to control the maximum number of results.

Streamlit UI

Install the optional UI extra:

pip install "evidence-fetcher[ui]"

Launch the Streamlit review app:

evidence-fetcher-ui

The UI uses the same Brave-backed retrieval, local reranking, relevance feedback, and explicit expansion workflow as the CLI.

Interactive relevance feedback:

evidence-fetcher "retrieval augmented generation evaluation" --interactive

Interactive mode performs the same initial search, then lets you mark displayed results, request a local rerank, or explicitly expand retrieval with a refined query. Ordinary text and JSON commands are unchanged and remain non-interactive. Interactive mode is text-only; combining --interactive with --format json is rejected.

Supported interactive commands:

more 1 3
less 4
irrelevant 5
unsure 2
show        # or rerank
expand
help        # or ?
done        # or quit, q, exit

Result numbers are presentation-only positions in the currently displayed list. Feedback is stored against each evidence URL, and after show the same number may refer to a different result. The interactive prompt shows the number of displayed results and accumulated feedback items, for example [10 results, 2 feedback items] >.

show reranks only the current candidate set using accumulated feedback. expand builds a deterministic lexical query from the original query and feedback-marked evidence, performs one additional Brave retrieval, merges any new URLs into the session, preserves accumulated feedback, and reranks the expanded candidate set. Expansion is always user-triggered; feedback commands do not automatically run new searches.

Example session:

$ evidence-fetcher "retrieval augmented generation evaluation" --interactive
Interactive relevance feedback. Type 'help' for commands.
1. Result title
   https://example.com/page
   Short snippet...
[10 results, 0 feedback items] > more 1
recorded more: 1 (1 feedback item total)
[10 results, 1 feedback item] > irrelevant 3
recorded irrelevant: 3 (2 feedback items total)
[10 results, 2 feedback items] > show
Updated ranking from accumulated feedback:
1. Result title
   https://example.com/page
   Short snippet...
[10 results, 2 feedback items] > expand
Searching with refined query:
retrieval augmented generation evaluation benchmark faithfulness
Added 4 new results.
Updated ranking:
1. Result title
   https://example.com/page
   Short snippet...
[14 results, 2 feedback items] > done

Architecture

The public API is intentionally small: fetch_evidence() accepts a query and returns normalized, deduplicated Evidence results. Provider-specific HTTP requests, credential handling, and response parsing live behind an internal Brave Search provider. The provider depends on a small HTTP client protocol, so tests and future integrations can inject a mock client instead of using the network.

Candidate retrieval and local relevance ranking are separate steps. Brave Search retrieves candidate evidence from the web; a replaceable local ranker then orders those candidates against the original query. The default ranker is a small standard-library TF-IDF implementation over evidence titles and snippets. This is the first foundation for Interactive Information Retrieval: the code now has a boundary where a workflow can collect user relevance feedback and rerank results.

Feedback-aware reranking is available through the Python/domain layer. A SearchSession retains the original query, candidate evidence, current ranked results, and accumulated RelevanceFeedback entries keyed by evidence URL. The TF-IDF ranker applies a lightweight Rocchio-style query update: it keeps the original query vector, moves it toward evidence marked MORE or RELEVANT, moves it away from evidence marked LESS or IRRELEVANT, and records UNSURE without changing ranking.

Feedback-informed retrieval expansion is separate from local reranking. The default query refiner tokenizes the original query and feedback-marked evidence, removes small stop words, scores candidate expansion terms lexically with TF-IDF, avoids duplicating original query terms, suppresses terms associated mainly with irrelevant evidence, and appends up to four useful terms. The refined query is printed before each expand retrieval. Expansion does not persist a session across invocations, run in the background, use an LLM, or send feedback anywhere except the explicit refined Brave query.

Package-specific exceptions distinguish configuration errors, request failures, and malformed provider responses.

Only Brave Search is implemented in this slice. No persistence or additional search providers are included yet.

IIR review caution

Interactive feedback can make evidence review faster, but it can also drift toward cherry-picking if the reviewer marks only convenient sources as useful. Treat reranking and expansion as prioritization aids. They do not replace balanced judgment, source quality checks, or deliberate review of evidence that complicates the initial hypothesis.

Development

This project uses uv for environment and dependency management.

uv sync
uv run pytest
uv run ruff check .
uv run mypy src
uv build
uv run sphinx-build -b html docs/source docs/built

Common shortcuts are available through make:

make check
make docs
make clean

Documentation

Documentation is built with Sphinx and Furo from docs/source/ into docs/built/. Read the Docs should build the documentation through its normal repository integration.

Releases

Releases are managed with Python Semantic Release and Conventional Commits. The GitHub Actions release workflow is intended to create the release commit, tag, GitHub release, wheel, sdist, and PyPI publication through trusted publishing.