mdsift 0.3.0

Sift a markdown workspace: BM25F field-weighted search, structural grep, and multi-layer graph exploration (PPR) for AI agents. Live-updating MCP server, no embeddings required.

mdsift

Sift a markdown workspace. Field-weighted keyword search, structural grep, and multi-layer graph exploration for AI agents — live-updating, embedding-free, exposed as an MCP server.

Point it at a folder of markdown. Your agent gets seven tools that stay fresh as files change, with sub-10 ms queries and zero model downloads.

Why not just BM25 + a wikilink graph?

Plenty of tools do flat BM25 over markdown. mdsift differs in four ways:

1. BM25F, not BM25. Tree-sitter parses every file into fields — title, headings, code identifiers, link text, body — and ranking weights them (a hit in a title or a function name beats a hit in prose). Code identifiers are subtokenized, so both refreshAccessToken and token match.
2. A multi-layer graph with agent-controlled weights. Three edge families — explicit links (markdown + wiki-links), folder hierarchy, and statistical TF-IDF similarity — queried together via Personalized PageRank. The agent dials per-edge-type multipliers per query: "related by links" vs "related by vocabulary" vs "related by location" are different questions.
3. Structural grep. Regex scoped to where in the document: only inside code blocks (optionally by language), only in headings, only in link text. Powered by ripgrep underneath, so it is fast and always index-fresh.
4. No embeddings. Pure lexical + graph statistics. Instant cold start, fully offline, no GPU, no model cache.

Install

pip install mdsift          # once published; from source: pip install -e .
# ripgrep must be on PATH:  apt install ripgrep | brew install ripgrep

Use

# MCP server (stdio) for Claude Desktop / Claude Code / any MCP client
mdsift-server /path/to/workspace

# CLI
mdsift /path/to/workspace search  '{"query": "auth retry", "top_n": 5}'
mdsift /path/to/workspace grep    '{"pattern": "retry.*backoff", "scope": "code", "lang": "python"}'
mdsift /path/to/workspace explore '{"seeds": {"docs/auth.md": 1.0}, "preset": "semantic"}'
mdsift /path/to/workspace clusters '{"min_size": 5}'

MCP client config:

{"mcpServers": {"mdsift": {"command": "mdsift-server", "args": ["/path/to/workspace"]}}}

Tools

Tool	Use when
search	Fuzzy/conceptual queries; BM25F over weighted fields. Hits include the best-matching section snippet, so agents often skip the full-file read. expand=true merges graph-related files.
read_section	Fetch one section by heading (fuzzy-matched) — the token-efficient follow-up to search.
grep	Exact strings / regex. scope=code|headings|body|links, lang filter, path glob/regex. Never stale.
deps	Direct neighborhood of one file: links in/out, folder relations, similarity edges.
explore	PPR from seed files. Presets structural / semantic / spatial / balanced, or explicit weights (0 prunes a layer).
clusters	Topic map: graph communities + TF-IDF labels (preset topics).
relate	Weighted shortest connection between two files.
status	Corpus, vocabulary, and edge counts.

Composition patterns agents use well: grep for candidates → search to rank them; search for seeds → explore to widen context; clusters on first contact with an unfamiliar workspace.

See docs/token-efficiency.md for an honest analysis of when this saves agent tokens — the short version: ranked results and PPR exploration replace multi-turn read loops, and section snippets attack the file-consumption side, where most tokens actually live.

All tools accept format (toon default — 45–60% fewer tokens on tabular results — or json), max_tokens (server-side response budget), and search accepts session (repeat-snippet dedup), granularity (file/section), tags (frontmatter filter), and recency (opt-in freshness boost). Frontmatter parsing covers Agent Skill (SKILL.md) files: multi-line block-scalar descriptions and nested metadata: are parsed, and skill name/description are indexed so skills are searchable by capability. Format coverage is tested against realistic CLAUDE.md (including @path imports, which become graph edges), AGENTS.md (root + nested per-directory), MADR and Nygard-style ADRs, and SKILL.md fixtures — see tests/test_agent_formats.py — plus the full GitHub Copilot customization surface (copilot-instructions.md, *.instructions.md/applyTo, *.prompt.md, *.chatmode.md, *.agent.md, .github/skills), Cursor .mdc rules (indexed by default), and Claude Rules — see tests/test_copilot_formats.py. Hidden directories (.github, .cursor, .claude) are scanned, grepped, and watched.

Performance

Measured on a synthetic 2,000-file workspace (python bench.py reproduces):

Path	Latency
Cold build	~2.5 s
search	p50 ≈ 3 ms
explore (PPR over 4 layers)	p50 ≈ 5 ms
grep scoped	p50 ≈ 45 ms (ripgrep-bound)
clusters cold / cached	~300 ms / ~3 ms
Live file update (parse → index → graph)	p50 ≈ 0.5 ms

How: columnar postings (int32 ids + uint16 tfs) with numpy-vectorized scoring and a delta/tombstone tier for updates; per-edge-type scipy CSR matrices summed at query time so agent-supplied weights cost microseconds; similarity edges maintained via rare-term candidate shortlisting (never all-pairs); folder nodes with capped sibling edges keep the graph linear. One index serves both ranking and similarity vectors.

Architecture

watchdog (200 ms debounce)
  └─ tree-sitter (block + inline grammars)
       ├─ BM25F index  ── search, similarity vectors
       ├─ span index   ── structural grep (via ripgrep --json)
       └─ graph        ── link / sibling / parent / similar layers
                          └─ PPR, clusters, shortest path

Update cost is O(changed file + shortlisted neighbors), never O(corpus).

Rust migration path

The data layout (interned ids, columnar postings, CSR layers) and the MCP JSON contract port 1:1 to Rust. First step if ever needed: move index.search and graph.ppr into a PyO3 extension; nothing else changes.

Development

pip install -e ".[dev]"
pytest tests/ -q      # 11 tests
python bench.py       # correctness + latency suite on 2,000 generated files

License

MIT. Dependencies: numpy/scipy (BSD-3), tree-sitter + grammar (MIT), watchdog (Apache-2.0); ripgrep (MIT/Unlicense) is invoked as a subprocess, not bundled.