Skip to main content

mdsift

Sift a markdown workspace. Field-weighted keyword search, structural grep, and multi-layer graph exploration for AI agents — live-updating, embedding-free, exposed as an MCP server.

Point it at a folder of markdown. Your agent gets twelve tools that stay fresh as files change, with sub-10 ms queries and zero model downloads.

Plenty of tools do flat BM25 over markdown. mdsift differs in four ways:

  1. BM25F, not BM25. Tree-sitter parses every file into fields — title, headings, code identifiers, link text, body — and ranking weights them (a hit in a title or a function name beats a hit in prose). Code identifiers are subtokenized, so both refreshAccessToken and token match.
  2. A multi-layer graph with agent-controlled weights. Three edge families — explicit links (markdown + wiki-links), folder hierarchy, and statistical TF-IDF similarity — queried together via Personalized PageRank. The agent dials per-edge-type multipliers per query: "related by links" vs "related by vocabulary" vs "related by location" are different questions.
  3. Structural grep. Regex scoped to where in the document: only inside code blocks (optionally by language), only in headings, only in link text. Powered by ripgrep underneath, so it is fast and always index-fresh.
  4. No embeddings. Pure lexical + graph statistics. Instant cold start, fully offline, no GPU, no model cache.

Install

pip install mdsift          # once published; from source: pip install -e .
# ripgrep must be on PATH:  apt install ripgrep | brew install ripgrep

Use

# MCP server (stdio) for Claude Desktop / Claude Code / any MCP client
mdsift-server /path/to/workspace

# multi-repo: several roots become logical workspaces; paths are "alias/rel"
mdsift-server frontend=~/code/web backend=~/code/api docs=~/code/docs

# CLI
mdsift /path/to/workspace search  '{"query": "auth retry", "top_n": 5}'
mdsift /path/to/workspace grep    '{"pattern": "retry.*backoff", "scope": "code", "lang": "python"}'
mdsift /path/to/workspace explore '{"seeds": {"docs/auth.md": 1.0}, "preset": "semantic"}'
mdsift /path/to/workspace clusters '{"min_size": 5}'

MCP client config:

{"mcpServers": {"mdsift": {"command": "mdsift-server", "args": ["/path/to/workspace"]}}}

Tools

Tool Use when
gather Start here. Viewpoint-aware (same question, audience-appropriate evidence). One call: topic map + best sections (extract-compressed) + graph-related files. Replaces the search→read→search gathering loop.
ontology Derived concept taxonomy: broader/narrower terms, full-corpus role-typed grounding with coverage stats, mined or curated definitions, bounded-context boundaries, cross-context synonym/variant alignment, non-markdown implementation grounding, glossary curation round-trip with mermaid export, and LLM-entity injection with AESOP-style reconciliation (docs/entity-injection.md).
trace Traceability over workspace IDs (REQ_/ADR-/CHK_ + custom families): coverage report with dangling/uncovered/orphan invariants, or resolve any ID both directions — definition, references by role, co-occurring IDs.
vocabulary The workspace's own language: shared keyphrases, code identifiers, acronyms. Also sampled into the initialize instructions (round-zero).
search Fuzzy/conceptual queries; viewpoint lens (stakeholder/data/operational/technical/security); BM25F over weighted fields. Hits include the best-matching section snippet, so agents often skip the full-file read. expand=true merges graph-related files.
read_section Fetch one section by heading (fuzzy-matched) — the token-efficient follow-up to search.
grep Exact strings / regex. scope=code|headings|body|links, lang filter, path glob/regex. Never stale.
deps Direct neighborhood of one file: links in/out, folder relations, similarity edges.
explore PPR from seed files. Presets structural / semantic / spatial / balanced, or explicit weights (0 prunes a layer).
clusters Topic map: graph communities + TF-IDF labels (preset topics).
relate Weighted shortest connection between two files.
status Corpus, vocabulary, and edge counts.

The server ships MCP instructions carrying a live workspace-vocabulary sample (ingested at round zero, before any tool call) and steering agents to gather → read_section with session ids and budgets, and search responses embed hints pointing to graph neighbors that lexical results missed. Composition patterns agents use well: grep for candidates → search to rank them; search for seeds → explore to widen context; clusters on first contact with an unfamiliar workspace.

See docs/token-efficiency.md for an honest analysis of when this saves agent tokens — the short version: ranked results and PPR exploration replace multi-turn read loops, and section snippets attack the file-consumption side, where most tokens actually live.

All tools accept format (toon default — 45–60% fewer tokens on tabular results — or json), max_tokens (server-side response budget), and search accepts session (repeat-snippet dedup), granularity (file/section), tags (frontmatter filter), and recency (opt-in freshness boost), and snippet_compress (light default strips markdown syntax from previews; extract keeps only query-relevant sentences — model-free extractive compression via the index's IDF). Frontmatter parsing covers Agent Skill (SKILL.md) files: multi-line block-scalar descriptions and nested metadata: are parsed, and skill name/description are indexed so skills are searchable by capability. Format coverage is tested against realistic CLAUDE.md (including @path imports, which become graph edges), AGENTS.md (root + nested per-directory), MADR and Nygard-style ADRs, and SKILL.md fixtures — see tests/test_agent_formats.py — plus the full GitHub Copilot customization surface (copilot-instructions.md, *.instructions.md/applyTo, *.prompt.md, *.chatmode.md, *.agent.md, .github/skills), Cursor .mdc rules (indexed by default), and Claude Rules — see tests/test_copilot_formats.py. Hidden directories (.github, .cursor, .claude) are scanned, grepped, and watched. Multi-repo mode gives every root a workspace alias: one shared graph (cross-repo similarity, clusters, and wiki-links), per-repo links stay repo-local, and path_glob="alias/**" scopes any tool to one repository.

Performance

Measured on a synthetic 2,000-file workspace (python bench.py reproduces):

Path Latency
Cold build (incl. vocabulary extraction) ~4.4 s
search (plain / with compressed snippets) p50 ≈ 3.5 / 5.0 ms
explore (PPR over 4 layers) p50 ≈ 5 ms
grep scoped p50 ≈ 40 ms (ripgrep-bound)
clusters cold / cached ~270 ms / ~4 ms
ontology build (cached; rescan only when concepts change) ~150–280 ms
viewpoint classification (cached, live-invalidated) ~40 ms
Live file update (parse → index → graph → vocab) p50 ≈ 0.7 ms

How: columnar postings (int32 ids + uint16 tfs) with numpy-vectorized scoring and a delta/tombstone tier for updates; per-edge-type scipy CSR matrices summed at query time so agent-supplied weights cost microseconds; similarity edges maintained via rare-term candidate shortlisting (never all-pairs); folder nodes with capped sibling edges keep the graph linear. One index serves both ranking and similarity vectors.

Architecture

watchdog (200 ms debounce, one observer per workspace root)
  └─ tree-sitter (block + inline grammars) + frontmatter parser
       ├─ BM25F index (files)     ── search, similarity vectors
       ├─ BM25F index (sections)  ── granularity="section"
       ├─ span index              ── structural grep, read_section, snippets
       ├─ vocabulary              ── RAKE phrases, acronyms, anchor terms
       └─ graph                   ── link / sibling / parent / similar layers
                                     └─ PPR (explore, gather), clusters, relate
Output: TOON encoding, snippet compression (light/extract), token budgets,
session dedup. Multi-root: paths are alias/rel over one shared graph.

Update cost is O(changed file + shortlisted neighbors), never O(corpus).

Rust migration path

The data layout (interned ids, columnar postings, CSR layers) and the MCP JSON contract port 1:1 to Rust. First step if ever needed: move index.search and graph.ppr into a PyO3 extension; nothing else changes.

Hook mode

mdsift-hook intercepts whole-file markdown reads (Claude Code PreToolUse; opencode plugin) and answers with the file's heading TOC plus scoped recovery moves — enforcement instead of guidance, effective even under skip-permissions. SessionStart injects vocabulary from the committed ontology snapshot. See docs/hook-mode.md.

Evaluation

eval/ contains a four-arm design for measuring the token effect on coding agents, including a deterministic generator for a realistic multi-repo fullstack workspace with full agentic harnesses and planted gradeable tasks. See eval/README.md.

Development

pip install -e ".[dev]"
pytest tests/ -q      # 69 tests
python bench.py       # correctness + latency suite on 2,000 generated files

License

MIT. Dependencies: numpy/scipy (BSD-3), tree-sitter + grammar (MIT), watchdog (Apache-2.0); ripgrep (MIT/Unlicense) is invoked as a subprocess, not bundled.

Metadata

Release files for mdsift 0.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for mdsift 0.4.0
File Size Uploaded Provenance Actions
mdsift-0.4.0.tar.gz 193.0 kB Without provenance Details for mdsift-0.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for mdsift 0.4.0
File Interpreter ABI Platform Actions
mdsift-0.4.0-py3-none-any.whl Python 3 none any Details for mdsift-0.4.0-py3-none-any.whl

Total release size: 259.9 kB

Release files / mdsift-0.4.0.tar.gz

Download URL mdsift-0.4.0.tar.gz
Size 193.0 kB
Tags Source
SHA-256 checksum
How to use checksums
ad6c58ab507a68621d4ac5a0ad852317ab464236d6f291b92cc053f4ed545bb6
BLAKE2b-256 checksum
How to use checksums
3f993b8c1ee0aac23ffceb7e47d2b89202b7ca293b866a711e93b19fb019c171
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.3

Release files / mdsift-0.4.0-py3-none-any.whl

Download URL mdsift-0.4.0-py3-none-any.whl
Size 66.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
178e0dab93db4a4f451c09cb81a570174855fcc39874018d2903091ca29a6685
BLAKE2b-256 checksum
How to use checksums
8857399fd07f6cb74c934cd90c2b7c306147298fffd30f4b271790d9c5eee61e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.12.3

Security for mdsift 0.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.4.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page