mdsift
Sift a markdown workspace. Field-weighted keyword search, structural grep, and multi-layer graph exploration for AI agents — live-updating, embedding-free, exposed as an MCP server.
Point it at a folder of markdown. Your agent gets twelve tools that stay fresh as files change, with sub-10 ms queries and zero model downloads.
Why not just BM25 + a wikilink graph?
Plenty of tools do flat BM25 over markdown. mdsift differs in four ways:
- BM25F, not BM25. Tree-sitter parses every file into fields — title,
headings, code identifiers, link text, body — and ranking weights them
(a hit in a title or a function name beats a hit in prose). Code
identifiers are subtokenized, so both
refreshAccessTokenandtokenmatch. - A multi-layer graph with agent-controlled weights. Three edge families — explicit links (markdown + wiki-links), folder hierarchy, and statistical TF-IDF similarity — queried together via Personalized PageRank. The agent dials per-edge-type multipliers per query: "related by links" vs "related by vocabulary" vs "related by location" are different questions.
- Structural grep. Regex scoped to where in the document: only inside code blocks (optionally by language), only in headings, only in link text. Powered by ripgrep underneath, so it is fast and always index-fresh.
- No embeddings. Pure lexical + graph statistics. Instant cold start, fully offline, no GPU, no model cache.
Install
pip install mdsift # once published; from source: pip install -e .
# ripgrep must be on PATH: apt install ripgrep | brew install ripgrep
Use
# MCP server (stdio) for Claude Desktop / Claude Code / any MCP client
mdsift-server /path/to/workspace
# multi-repo: several roots become logical workspaces; paths are "alias/rel"
mdsift-server frontend=~/code/web backend=~/code/api docs=~/code/docs
# CLI
mdsift /path/to/workspace search '{"query": "auth retry", "top_n": 5}'
mdsift /path/to/workspace grep '{"pattern": "retry.*backoff", "scope": "code", "lang": "python"}'
mdsift /path/to/workspace explore '{"seeds": {"docs/auth.md": 1.0}, "preset": "semantic"}'
mdsift /path/to/workspace clusters '{"min_size": 5}'
MCP client config:
{"mcpServers": {"mdsift": {"command": "mdsift-server", "args": ["/path/to/workspace"]}}}
Tools
| Tool | Use when |
|---|---|
gather |
Start here. Viewpoint-aware (same question, audience-appropriate evidence). One call: topic map + best sections (extract-compressed) + graph-related files. Replaces the search→read→search gathering loop. |
ontology |
Derived concept taxonomy: broader/narrower terms, full-corpus role-typed grounding with coverage stats, mined or curated definitions, bounded-context boundaries, cross-context synonym/variant alignment, non-markdown implementation grounding, glossary curation round-trip with mermaid export, and LLM-entity injection with AESOP-style reconciliation (docs/entity-injection.md). |
trace |
Traceability over workspace IDs (REQ_/ADR-/CHK_ + custom families): coverage report with dangling/uncovered/orphan invariants, or resolve any ID both directions — definition, references by role, co-occurring IDs. |
vocabulary |
The workspace's own language: shared keyphrases, code identifiers, acronyms. Also sampled into the initialize instructions (round-zero). |
search |
Fuzzy/conceptual queries; viewpoint lens (stakeholder/data/operational/technical/security); BM25F over weighted fields. Hits include the best-matching section snippet, so agents often skip the full-file read. expand=true merges graph-related files. |
read_section |
Fetch one section by heading (fuzzy-matched) — the token-efficient follow-up to search. |
grep |
Exact strings / regex. scope=code|headings|body|links, lang filter, path glob/regex. Never stale. |
deps |
Direct neighborhood of one file: links in/out, folder relations, similarity edges. |
explore |
PPR from seed files. Presets structural / semantic / spatial / balanced, or explicit weights (0 prunes a layer). |
clusters |
Topic map: graph communities + TF-IDF labels (preset topics). |
relate |
Weighted shortest connection between two files. |
status |
Corpus, vocabulary, and edge counts. |
The server ships MCP instructions carrying a live workspace-vocabulary sample (ingested at round zero, before any tool call) and steering agents to gather → read_section with session ids and budgets, and search responses embed hints pointing to graph neighbors that lexical results missed. Composition patterns agents use well: grep for candidates → search to rank
them; search for seeds → explore to widen context; clusters on first
contact with an unfamiliar workspace.
See docs/token-efficiency.md for an honest analysis of when this saves agent tokens — the short version: ranked results and PPR exploration replace multi-turn read loops, and section snippets attack the file-consumption side, where most tokens actually live.
All tools accept format (toon default — 45–60% fewer tokens on tabular
results — or json), max_tokens (server-side response budget), and search
accepts session (repeat-snippet dedup), granularity (file/section),
tags (frontmatter filter), and recency (opt-in freshness boost), and snippet_compress (light default strips markdown syntax from previews; extract keeps only query-relevant sentences — model-free extractive compression via the index's IDF).
Frontmatter parsing covers Agent Skill (SKILL.md) files: multi-line block-scalar
descriptions and nested metadata: are parsed, and skill name/description
are indexed so skills are searchable by capability. Format coverage is
tested against realistic CLAUDE.md (including @path imports, which become
graph edges), AGENTS.md (root + nested per-directory), MADR and Nygard-style
ADRs, and SKILL.md fixtures — see tests/test_agent_formats.py — plus the
full GitHub Copilot customization surface (copilot-instructions.md,
*.instructions.md/applyTo, *.prompt.md, *.chatmode.md, *.agent.md,
.github/skills), Cursor .mdc rules (indexed by default), and Claude Rules —
see tests/test_copilot_formats.py. Hidden directories (.github, .cursor,
.claude) are scanned, grepped, and watched. Multi-repo mode gives every root
a workspace alias: one shared graph (cross-repo similarity, clusters, and
wiki-links), per-repo links stay repo-local, and path_glob="alias/**"
scopes any tool to one repository.
Performance
Measured on a synthetic 2,000-file workspace (python bench.py reproduces):
| Path | Latency |
|---|---|
| Cold build (incl. vocabulary extraction) | ~4.4 s |
search (plain / with compressed snippets) |
p50 ≈ 3.5 / 5.0 ms |
explore (PPR over 4 layers) |
p50 ≈ 5 ms |
grep scoped |
p50 ≈ 40 ms (ripgrep-bound) |
clusters cold / cached |
~270 ms / ~4 ms |
ontology build (cached; rescan only when concepts change) |
~150–280 ms |
| viewpoint classification (cached, live-invalidated) | ~40 ms |
| Live file update (parse → index → graph → vocab) | p50 ≈ 0.7 ms |
How: columnar postings (int32 ids + uint16 tfs) with numpy-vectorized
scoring and a delta/tombstone tier for updates; per-edge-type scipy CSR
matrices summed at query time so agent-supplied weights cost microseconds;
similarity edges maintained via rare-term candidate shortlisting (never
all-pairs); folder nodes with capped sibling edges keep the graph linear.
One index serves both ranking and similarity vectors.
Architecture
watchdog (200 ms debounce, one observer per workspace root)
└─ tree-sitter (block + inline grammars) + frontmatter parser
├─ BM25F index (files) ── search, similarity vectors
├─ BM25F index (sections) ── granularity="section"
├─ span index ── structural grep, read_section, snippets
├─ vocabulary ── RAKE phrases, acronyms, anchor terms
└─ graph ── link / sibling / parent / similar layers
└─ PPR (explore, gather), clusters, relate
Output: TOON encoding, snippet compression (light/extract), token budgets,
session dedup. Multi-root: paths are alias/rel over one shared graph.
Update cost is O(changed file + shortlisted neighbors), never O(corpus).
Rust migration path
The data layout (interned ids, columnar postings, CSR layers) and the MCP JSON
contract port 1:1 to Rust. First step if ever needed: move index.search and
graph.ppr into a PyO3 extension; nothing else changes.
Hook mode
mdsift-hook intercepts whole-file markdown reads (Claude Code
PreToolUse; opencode plugin) and answers with the file's heading TOC plus
scoped recovery moves — enforcement instead of guidance, effective even
under skip-permissions. SessionStart injects vocabulary from the committed
ontology snapshot. See docs/hook-mode.md.
Evaluation
eval/ contains a four-arm design for measuring the token effect on coding
agents, including a deterministic generator for a realistic multi-repo
fullstack workspace with full agentic harnesses and planted gradeable tasks.
See eval/README.md.
Development
pip install -e ".[dev]"
pytest tests/ -q # 69 tests
python bench.py # correctness + latency suite on 2,000 generated files
License
MIT. Dependencies: numpy/scipy (BSD-3), tree-sitter + grammar (MIT), watchdog (Apache-2.0); ripgrep (MIT/Unlicense) is invoked as a subprocess, not bundled.
Metadata
Release files for mdsift 0.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| mdsift-0.4.0.tar.gz | 193.0 kB | Without provenance | Details for mdsift-0.4.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| mdsift-0.4.0-py3-none-any.whl | Python 3 | none | any | Details for mdsift-0.4.0-py3-none-any.whl |
Total release size: 259.9 kB
Release files / mdsift-0.4.0.tar.gz
| Download URL | mdsift-0.4.0.tar.gz |
|---|---|
| Size | 193.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ad6c58ab507a68621d4ac5a0ad852317ab464236d6f291b92cc053f4ed545bb6
|
|
BLAKE2b-256 checksum How to use checksums |
3f993b8c1ee0aac23ffceb7e47d2b89202b7ca293b866a711e93b19fb019c171
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
Release files / mdsift-0.4.0-py3-none-any.whl
| Download URL | mdsift-0.4.0-py3-none-any.whl |
|---|---|
| Size | 66.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
178e0dab93db4a4f451c09cb81a570174855fcc39874018d2903091ca29a6685
|
|
BLAKE2b-256 checksum How to use checksums |
8857399fd07f6cb74c934cd90c2b7c306147298fffd30f4b271790d9c5eee61e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.12.3
|
Security for mdsift 0.4.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.