asset-aware-mcp

Citation-ready document infrastructure for AI agents: turn PDFs, DOCX files, tables, figures, and evidence spans into reusable assets and Foam/LightRAG wikis.

🌐 繁體中文 · Docs Site · GitHub Wiki

🎯 Why Asset-Aware MCP?

AI cannot directly read image files on your computer. This is a common misconception.

Real-world Effect

# After retrieving the image via MCP, the AI can analyze it directly:

User: What is this figure about?

AI: This is the architecture diagram for Scaled Dot-Product Attention:
    1. Inputs: Q (Query), K (Key), V (Value)
    2. MatMul of Q and K
    3. Scale (1/√dₖ)
    4. Optional Mask (for decoder)
    5. SoftMax normalization
    6. Final MatMul with V to get the output

This is the value of Asset-Aware MCP - enabling AI Agents to truly "see" and understand charts and tables in your PDF literature.

✨ Features

• 📄 Asset-Aware ETL - PDF → Markdown with a pluggable multi-engine parser (ETL_ENGINE):
  • PyMuPDF (default) - Fast extraction (~50MB), no models required
  • PyMuPDF4LLM ([pdf-plus]) - Drop-in layout-aware upgrade, no GPU
  • Docling ([docling]) - MIT-licensed layout+table+formula+chart engine; bridges through an isolated .venv-docling interpreter when the main environment can't install it directly (see docs/docling-setup.md)
  • MinerU - Adapter retained, but the packaged extra is on security hold while MinerU pins a vulnerable transformers<5 chain
  • Marker (use_marker=True) - High-precision structured parsing code path retained, but packaged runtime remains on security hold until upstream marker-pdf supports patched Pillow
• 🧩 Unified Segmentation Export - Normalized segmentation.json merges manifest, blocks, reading order, and persisted markdown line spans for downstream tools and extensions.
• 🩺 Safe PDF Preflight Router - document(op="preflight") classifies each page as native, sparse, image, scanned, or hybrid; returns 1-based top-left locators, source SHA-256, OCR reasons, and a bounded extraction-engine recommendation from a process-isolated inspector.
• 📦 Reusable Agent Asset Bundles - document(op="export_assets") writes deterministic manifest.json, assets.jsonl, copied media, and a portable Foam index.md/notes/** subtree while preserving stable IDs, hashes, locators, and citation refs.
• 🛡️ PDF Safety/Structure/Coverage/Accessibility Audits - OpenDataloader-inspired artifact-only reports flag suspicious hidden/off-page/prompt-injection text, native structure signals, segmentation coverage gaps, and accessibility/readability readiness via the existing document facade. document(op="prepare_ai") and document(op="auto") expose agent-ready status and next actions without adding public tools.
• 🧭 Structural Pointer Retrieval - Proxy-Pointer-inspired document(op="pointer_index"), document(op="structural_retrieve"), and document(op="compare") preserve section breadcrumbs, line/char/byte locators, source hashes, asset IDs, and evidence-span provenance without adding MCP tools.
• 🖼️ Layout Overlay Debugging - Render page overlays from original.pdf to inspect bbox, segment type, and reading order visually.
• 🔤 On-Demand OCR Preprocessing - Optional ocrmypdf preprocessing path for scanned PDFs before ETL.
• 🧭 Section Navigation - Dynamic hierarchy section tree through the section facade: browse, search, detail, content reading, and block extraction for any depth of headings.
• 🔄 Async Job Pipeline - Supports asynchronous ingest, configured structured parse, OCR, and conversion jobs with progress tracking.
• 🔀 Mixed-Format Batch Ingestion - document(op="auto", file_paths=[...]) auto-detects a batch mixing PDF with DOCX/DOC/ODT/ODS, ingests each file through its correct existing engine in one background job, isolates per-file failures so one bad file cannot abort the rest, and reports per-file progress — no new public tool required.
• 🗺️ Document Manifest - Provides a structured "map" of the document for precise data access by Agents.
• 🧠 LightRAG Integration - Knowledge Graph + Vector Index, supporting cross-document comparison and reasoning.
• 🧾 Verified Citation Bundles - citation_bundle, Foam evidence packs, citation health checks, table/figure evidence notes, and claim promotion export citation-ready spans with locator, quote/hash, context, CRAAP scaffold, and verification status.
• 📝 Docx Editing (DFM) - Edit .docx files in Markdown via Docx-Flavored Markdown format. Supports legacy .doc, .odt, and .ods ingest via LibreOffice auto-conversion. The balanced surface keeps 6 DOCX/DFM public entrypoints for ingest, read, save, validation, conversion, table edit planning, and Docx ↔ A2T bridges.
• 🛡️ DFM Integrity Checker - Automatic validation and auto-repair at every pipeline stage (post-ingest, pre-save, post-save). Catches orphan markers, column mismatches, and format inconsistencies.
• 📊 A2T (Anything to Table) - 7 operation-based tools for building professional tables from any source (PDF assets, Knowledge Graph, URLs, user input). Features: stable row IDs, row search/filter/paging, citation coverage, artifact-only large-table render, skipped-large-table UX, Citations (AssetRef), Audit Trail, Schema Evolution, Templates, Drafting, and Token-efficient resumption.
• 🖥️ VS Code Management Extension - Graphical interface for monitoring server status, ingested documents, document artifacts, citation spans, and A2T tables/drafts with one-click Excel export.
• 🔌 MCP SDK 2 Server - Uses the official Python SDK MCPServer API, runtime-injected context, and v2 clients. MCP SDK v1 is intentionally unsupported.
• 🏥 Medical Research Focus - Optimized for medical literature, supporting Base64 image transmission for Vision AI analysis.

🏗️ Architecture

┌─────────────────────────────────────────────────────────┐
│                    AI Agent (Copilot)                   │
└─────────────────────┬───────────────────────────────────┘
                      │ MCP Protocol (Tools & Resources)
┌─────────────────────▼───────────────────────────────────┐
│            MCP Server (Modular Presentation)            │
│  ┌─────────────────────────────────────────────────┐   │
│  │ tools/: 30 public tools (balanced surface)                   │   │
│  │   17 facade tools + 13 high-frequency shortcuts       │   │
│  │   compact=17 │ legacy/direct compatibility=63 │
│  └─────────────────────────────────────────────────┘   │
│  ┌─────────────────────────────────────────────────┐   │
│  │ resources/: 13 resources in 2 modules           │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│                  ETL Pipeline (DDD)                     │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │ PyMuPDF  │  │  Asset   │  │ LightRAG │              │
│  │ Adapter  │→ │  Parser  │→ │  Index   │              │
│  └──────────┘  └──────────┘  └──────────┘              │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│                   Local Storage                         │
│  ./data/                                                │
│  ├── {doc_id}/        # PDF document artifacts          │
│  ├── docx_{id}/       # Docx IR + DFM + Assets          │
│  ├── tables/          # A2T Tables (JSON/MD/XLSX)       │
│  │   └── drafts/      # Table Drafts (Persistence)      │
│  └── lightrag_db/     # Knowledge Graph                 │
└─────────────────────────────────────────────────────────┘

📁 Project Structure (DDD)

asset-aware-mcp/
├── src/
│   ├── domain/              # 🔵 Domain: Entities, Value Objects, Interfaces
│   ├── application/         # 🟢 Application: Doc Service, Table Service (A2T), Asset Service
│   ├── infrastructure/      # 🟠 Infrastructure: PyMuPDF, LightRAG, Excel Renderer
│   └── presentation/        # 🔴 Presentation: MCP SDK 2 MCPServer
├── data/                    # Document and Asset Storage
├── docs/
│   └── spec.md              # Technical Specification
├── tests/                   # Unit and Integration Tests
├── vscode-extension/        # VS Code Management Extension
└── pyproject.toml           # uv Project Config

📐 Architecture Diagrams

Visual overview for the project. All diagrams use consistent GitHub README style.

💡 All generation prompts are saved in docs/diagrams/ALL-PROMPTS.md for style consistency and regeneration.

🚀 Quick Start

# Install dependencies (using uv) — default install stays on the fast PyMuPDF backend
uv sync

# Optional high-fidelity PDF->asset engines:
# uv sync --extra pdf-plus   # PyMuPDF4LLM: drop-in layout-aware upgrade
# uv sync --extra docling    # Docling: MIT layout+table+formula+chart engine
# MinerU and Marker packaged extras are temporarily empty security holds.
# Then set ETL_ENGINE=pymupdf4llm|docling.

# Run MCP Server
uv run python -m src.presentation.server

# Or use the VS Code extension for graphical management

Runtime note: The VS Code extension prefers a managed Python 3.11 runtime when launching the MCP server via version-pinned uv tool run, with Python 3.10 fallback for older machines. This avoids native package builds on end-user machines, especially macOS systems without Xcode Command Line Tools, while keeping the project itself compatible with newer Python versions.

Installation scope note:

• The VS Code extension installs once per user (global). MCP launch env defaults DATA_DIR to workspace ./data and UV_CACHE_DIR to DATA_DIR/.uv-cache; Prepare Server Runtime warms a workspace .uv-cache, falling back to extension global storage only when no workspace is open.
• Runtime data stays with your repo: .env and assetAwareMcp.dataDir default to ./data, so ingested assets and the uv cache used by the launched server remain scoped to the current workspace.

Engine selection note: ETL_ENGINE picks the extraction backend (default pymupdf). The active packaged structured engines (pymupdf4llm, docling) lazy-load and gracefully fall back to PyMuPDF when their extra is not installed. Marker remains on hold because marker-pdf requires Pillow<11; MinerU is also on hold because MinerU 3.4.4 pins transformers<5 while current security fixes require transformers>=5.5. Both adapters remain in-tree, but this package will not install a known-vulnerable dependency chain. Use document(op="preflight", pdf_path="...") to choose between fast native extraction, OCR, and Docling before ingest.

Agent asset / Foam handoff:

document(op="preflight", pdf_path="/papers/source.pdf")
document(op="auto", file_paths=["/papers/source.pdf"])
document(op="export_assets", doc_id="doc_...", output_dir="agent-assets")

The exported directory is deterministic and portable: manifest.json is the bundle contract, assets.jsonl is the agent-readable inventory, and index.md plus notes/** can be mounted or copied into a Foam workspace.

🔌 MCP Tools

The default runtime surface is balanced: 30 public tools that keep the full document workflow available without overwhelming agents. It is made of 17 operation-based facade tools plus 13 high-frequency shortcuts. Set ASSET_AWARE_MCP_TOOL_SURFACE=compact for the 17 facade-only surface, or ASSET_AWARE_MCP_TOOL_SURFACE=legacy / ASSET_AWARE_MCP_ENABLE_LEGACY_TOOLS=true for the full 63-tool compatibility inventory.

See MCP Tools and Tool Consolidation for operation details, shortcut rationale, and legacy direct-tool mapping.

Agent handoff note: Use document(op="auto", file_paths=[...]) for new PDFs and document(op="auto", doc_id="...") or document(op="prepare_ai", doc_id="...") for existing documents. document(op="prepare_ai", output_format="json") returns the v2 readiness contract with status, blockers, warnings, capabilities, artifacts, missing_audits, invalid_audits, audit_artifacts, and next_actions. document(op="audit", doc_id="...") reuses current audit artifacts only when they are present and valid; pass refresh=true to rebuild safety, native-structure, coverage, and accessibility reports. Use document(op="pointer_index"), document(op="structural_retrieve", query="..."), and document(op="compare", doc_b_id="...", criteria="...") when an agent needs section-level structural retrieval or comparison without new public tools. Readiness and job-status artifact discovery are read-only, so status checks do not create document directories.

PDF audit caveat: The audit reports are inspired by OpenDataloader-style artifact workflows, but they are not a sanitizer, a PDF/UA certification, or an OpenDataloader compatibility layer. They preserve source artifacts and report conservative diagnostics for review.

🔧 Tech Stack

📋 Documentation

Installation guidance:

• Default install: uv sync (slim ~227 MB; no LightRAG/KG dependencies).

• LightRAG / Knowledge Graph backend (optional, since v0.6.34): uv tool install --upgrade --python 3.11 'asset-aware-mcp[lightrag]' for uvx/published users, or uv sync --extra lightrag for local source checkouts. Required before setting ENABLE_LIGHTRAG=true.

• VS Code extension: run the command Asset-Aware MCP: Install LightRAG Backend from the Command Palette; it auto-detects source vs published mode and emits the matching install command.

• OpenRouter optional preset (since v0.6.35): set LLM_BACKEND=openrouter, OPENROUTER_API_KEY=..., and optionally OPENROUTER_MODEL=liquid/lfm-2.5-1.2b-instruct:free for fast low-cost summaries and draft RAG answers. LightRAG retrieval still uses the configured embedding backend.

• High-fidelity PDF engines: uv sync --extra pdf-plus (PyMuPDF4LLM) or uv sync --extra docling (Docling), then set ETL_ENGINE accordingly. Docling ships a cross-platform isolated installer; see docs/docling-setup.md.

• MinerU and Marker backends: their adapters remain available for upstream testing, but the packaged extras are empty security holds until their dependency caps permit patched transformers and Pillow releases.

• VS Code extension: assetAwareMcp.enableMarkerBackend is retained as a setting, but the launcher will not install marker-pdf while the security hold is active.

• Technical Spec - Detailed technical specification

• Architecture - System architecture

• Constitution - Project principles

• Competitive Analysis - MCP + DOCX ecosystem landscape

📄 License

Apache License 2.0