net-sift 0.3.1

Net-Sift sweeps many sources for everything being said about a topic, ranks it, removes duplicates, and reports what it could not reach. The gap report is part of the answer, not an afterthought. Login-walled platforms are reached through your own logged-in Chromium browser, so there is no cookie copying and no private-API reverse engineering.

Table of contents

• Why Net-Sift
• Features
• How it works
• Install
• Quickstart
• Walled platforms
• Sources
• MCP tools
• Configuration
• Sessions and privacy
• Development
• Contributing
• Security
• License
• Acknowledgements

Why Net-Sift

Most research tools return a handful of links or a synthesized answer. Net-Sift returns a corpus with an explicit account of coverage: what was retrieved, what hit a ceiling, and what was out of reach. It is built for topic surveys, sentiment sweeps, competitor and discourse monitoring, and literature-style scans where a single search would under-serve the answer.

Features

• Many sources in one sweep, keyless by default (Bluesky, Hacker News, GitHub, arXiv, Polymarket, StockTwits, Mastodon, and more).
• Walled platforms (Twitter/X, Reddit, Instagram, Facebook, Bilibili, Xiaohongshu) through your logged-in Chromium browser via OpenCLI.
• Relevance ranking with head-entity grounding, CJK-aware tokenization, a recency boost, and engagement weighting.
• A gap-closing driver that bisects the time window to recover tails a source would otherwise hide.
• Per-source near-duplicate collapse, so repeats do not inflate the corpus while cross-platform coverage is preserved.
• Context-lean by design: the agent receives summaries, coverage, gaps, and the top results, never the raw corpus.
• Saved sessions you can continue later, deleted only on your confirmation.

How it works

query
  |
  v
sources (keyless)          access (walled, via OpenCLI + your browser)
  \_________________  _________________/
                    \/
         gap-closing driver (bisect on ceiling)
                    |
            dedup + ranking
                    |
         session on disk  --->  summary + coverage + gaps  --->  agent

Keyless sources call public endpoints directly. Walled platforms are reached by shelling to opencli <site> <command> against your logged-in browser. Results are normalized to one record shape, deduplicated, ranked, and written to a session. Only the summary returns to the caller.

Install

Install from source (not yet on PyPI):

uv tool install git+https://github.com/ali-rajabpour/Net-Sift.git
# or: pipx install git+https://github.com/ali-rajabpour/Net-Sift.git

Then run the setup wizard:

net-sift install              # guided: clients, OpenCLI, browser; add --yes for unattended

net-sift install is a wizard. It registers the MCP server and status bar with the clients it finds (Claude Code, Codex), installs OpenCLI automatically through npm with your consent, and walks you through connecting a Chromium browser, rechecking as it goes. Each step asks before it changes anything. Restart your client afterward so it picks up the server.

Quickstart

net-sift doctor               # what can be reached right now
net-sift search "topic" --platforms github,arxiv --max 50

In an MCP client, call deep_search:

deep_search(query="post-quantum cryptography adoption", since="2026-01-01")

You get a summary with per-source counts, a coverage map, the gap list, the top ranked items, and the corpus path.

Walled platforms

Twitter/X, Reddit, Instagram, Facebook, Bilibili, and Xiaohongshu need a logged-in Chromium browser and OpenCLI. See net_sift/guides/setup-opencli.md. Short version:

1. Install Node 20.18.1+ and @jackwener/opencli (or OpenCLIApp).
2. Use a Chromium browser (Chrome, Edge, Brave, Arc, Comet, and so on) with the OpenCLI Browser Bridge extension. Safari and Firefox cannot load it.
3. Log into the platforms you want in that browser.
4. Run net-sift doctor to confirm.

Desktop only. There is no headless or server path for walled platforms.

Sources

Full table in net_sift/guides/sources.md. Keyless sources are on by default; walled sources appear once OpenCLI is connected.

MCP tools

Configuration

All configuration is read from the environment. Net-Sift stores no credentials.

Sessions and privacy

Each search is written to ~/.net-sift/sessions/<id>/ as corpus.jsonl and meta.json. Nothing is sent anywhere. After a search, Net-Sift reminds you the corpus is kept so you can resume it, and deletes it only when you call cleanup. Walled platforms use your own browser session; Net-Sift never reads or copies your cookies.

Development

git clone https://github.com/ali-rajabpour/Net-Sift.git
cd Net-Sift
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
ruff check net_sift tests && pytest

See docs/architecture.md for the design.

Contributing

See CONTRIBUTING.md. Net-Sift is read-only research: it does not post, comment, or bypass authentication, and contributions stay within that scope.

Security

See SECURITY.md. Report vulnerabilities privately through GitHub.

License

GNU AGPL-3.0-or-later. See LICENSE. If you run a modified version as a network service, the AGPL requires you to offer users its source. This project includes code adapted from Agent Reach (MIT, a permissive license compatible with AGPL); that attribution is kept in NOTICE.

Acknowledgements

• OpenCLI for logged-in browser access.
• Agent Reach for the onboarding and doctor patterns.