mdcx

Convert a document collection to verified Markdown, package it into a single encrypted file, and query it from an agent through the Model Context Protocol.

Contents

• Overview
• Requirements
• Installation
• Quick start
• Conversion
• Packaging and querying
• Sent and received
• Working incrementally
• MCP server
• Language support
• Cross-language retrieval
• Portable paths
• Signing
• Encryption
• Limitations
• Tests
• Contributing
• Security
• Releases
• Authorship
• Citation
• Licence

Overview

mdcx converts a collection of documents to Markdown, verifies each conversion against its original, packages the corpus with its index and provenance into a single encrypted file, and serves that file to agents over the Model Context Protocol.

It addresses one constraint. An agent asked a question about a document collection must either receive the documents in its context window, which is bounded in size and billed per token, or query a component that holds an index and returns only the passages that bear on the question. mdcx implements the second. Three properties distinguish it from an extraction script:

• Fidelity is measured, not assumed. Every conversion is checked against the text the original exposes, read by a library independent of the engine that produced the conversion, and the coverage achieved is recorded per file.
• The corpus is a single encrypted artefact. Passages, index and provenance are held in one AES-256-GCM file whose header can be read without the key.
• Every passage carries its source. An answer can be cited against a document and a location rather than recalled.

Pipeline

Conversion. Each document is attempted by the least expensive engine capable of reading it and escalated only where that engine falls short: direct text extraction, then a pass that recovers the tables a page draws, then full layout analysis. Documents exposing no text are read by optical character recognition. Content the selected engine omitted is appended verbatim rather than reported as lost.

Over the collection used during development — 99 documents, 1,144,553 reference tokens — 594 tokens were not recovered, a coverage of 99.948%. Of the 95 documents that expose text, 70 were recovered in full and none fell below 99.5%. The remaining four are scanned drawings holding no text in the file; they are marked unverifiable, as no text original exists to measure them against.

Packaging. The corpus, its search index and the provenance of every passage are written to a single .mdcx file. The development collection produced 3.9 MB from 8.8 MB of Markdown. A growing collection is not rebuilt from the start: vectors already computed are reused, and a corpus exceeding what can be decrypted into memory is held as several packages queried as one.

Retrieval. A query returns the passages that answer it, each with its source document and its position in the ranking. Word matching and dense retrieval are merged by reciprocal rank, so a query reaches a document whether it shares that document's vocabulary or only its subject, including where the two are written in different languages. Over a corpus of 136 documents in 34 languages, the merged engines rank the expected document first for 135 of the 136 queries. Where no document in the corpus is about the question, the reply states this rather than presenting its nearest passage as an answer.

Measured cost

One query over the development collection — 99 documents, 180 MB — counted with the cl100k_base tokenizer:

The 435 model tokens comprise 20 for the question, 274 for the retrieved passage and 141 for the answer.

Reading the originals costs the whole collection because a PDF is a binary format: absent prior conversion there is no way to determine which of the 99 documents holds the answer, so all of them are extracted and read.

This is a single measurement, not an average, and the saving depends on how much text an answer requires. The work is not eliminated but relocated, from the context window, which is billed and finite, to local processing, which is neither. The local column rises for that reason.

Requirements

Python 3.11 or later. No other component is required to query a package.

The floor is 3.11 because a package is held as one SQLite database and serialised in memory to be encrypted, and sqlite3 gained the call that does so in that version. Earlier interpreters were declared supported and were not: neither building a package nor opening one worked there. Conversion and cross-language retrieval each add dependencies, listed under Installation.

Installation

Querying and conversion are separated because their requirements differ by two orders of magnitude.

Conversion accounts for the heavy dependencies. A recipient who only queries an .mdcx file installs neither Docling nor PyTorch.

The multilingual extra is required for queries that cross languages. Most of its size is the embedding model, downloaded once on first use. A single-language corpus does not require it.

The tables extra covers what a page does not draw. Tables in printed material are usually found from the rules drawn around them, which costs nothing and needs no extra; borderless ones — a screenshot of a spreadsheet, a layout held together by alignment — are read by a small model that reports where the rows and columns run. It reads the shape only: the words still come from the text layer of the document, so a cell cannot hold anything the page does not say. Without it those pages are read by Docling instead, which is slower but already present in the convert extra.

Quick start

pip install "mdcx[convert]"

mdcx-convert --input ./Documents --output ./Documents_md
mdcx pack --output ./Documents_md --target corpus.mdcx --key "passphrase"
mdcx search corpus.mdcx "where is the storage temperature stated" --key "passphrase"

Conversion

mdcx-convert --input ./Documents --output ./Documents_md

The output mirrors the input directory structure, adds a global index, and records for each file the coverage achieved against its original.

Supported formats

PDF, EPUB, Word, Excel, PowerPoint, HTML, Markdown, CSV and plain text.

The format of a file is determined from its first bytes rather than from its extension. Repositories are known to serve EPUB files from URLs ending in .pdf and declaring application/pdf, where only the content identifies the format correctly. Routing such a file by extension sends it to a reader that cannot open it, and the resulting failure is indistinguishable from a damaged document.

Plain text carries no signature, so its extension determines the format. A file whose content identifies no known format is skipped rather than assumed.

How much of the machine it uses

Converting a library is the heaviest thing this package does, and it runs on a machine somebody is working at. Nothing here is a constant: every figure is derived from the machine it finds, because the same number cannot be right on four processors and on thirty-two.

mdcx-convert --input ./Documents --output ./Documents_md --max-cores 4

Processors. A fifth are left free and the rest are used: nine on twelve, three on four. The cap is a budget for the whole run rather than a grant to each process, so it is divided among the workers and each is told its share. Without that division the structured engine asks for four threads of its own and eight workers ask for thirty-two on a machine of twelve, spending inside the pool the share that was carefully left outside it.

The card. How many processes may use it at once is decided by three ceilings, the smallest winning: the free video memory divided by what a worker holds with a full batch; how much of the material actually needs a model, which is the documents that expose no text and have nothing to extract; and leaving something for the processor. All three are needed. Without the first, asking for more workers by hand does the opposite of what it looks like — twelve on a 6 GB card ask for 15.7 GB and measured three times slower than three. Without the last, a large card takes every worker and leaves one for the bulk of the work, which is processor work.

That limit is then held by a gate every worker shares, taken around the model call rather than around the document, so a process reading text is not occupying a place on the card while it does.

The batch. What a page costs falls with the number of pages it travels with — 150 ms sending one, 75 with eight, 46 with twenty-four — so the batch is as large as the card allows once every worker is seated on it, and no larger. It is decided where both halves are known, because how much of the card a worker may hold depends on how many may hold it; a worker deciding for itself reads the free memory as though nobody else would.

Lanes. Documents are dispatched to two of them. Both may reach every engine and both are counted against the same limit on the card: the lane decides what is worth dispatching where, not what a document is allowed to reach. The lane used to decide both, which meant that moving a document out of the crowded lane also took away its structured engine.

Every one of these can be overridden — --max-cores, --gpu-workers, --cpu-workers, and MDCX_TATR_BATCH — and the derived figure is the default rather than a ruling. A machine that measures differently says so.

Checking a conversion before packaging

mdcx-search searches the converted Markdown directly, before there is a package, and quotes each passage with the document and pseudopath it came from. It is how a conversion is inspected while the folder is still open to correction.

mdcx-search "movable type" --output ./Documents_md
mdcx-search --phrases ./questions.txt --output ./Documents_md --json found.json

Passages are ranked with BM25 aggregated per document rather than in isolation, so a long document that covers a subject across several fragments is not beaten by a short unrelated one that repeats a term. --literal requires the exact phrase and nothing else; --bm25 ranks by relevance without literal matching.

This engine reads the Markdown folder. Retrieval over a built package, with meaning and across languages, is mdcx search and the MCP server.

Packaging and querying

mdcx pack --output ./Documents_md --target corpus.mdcx --key "passphrase"
mdcx info corpus.mdcx
mdcx search corpus.mdcx "where is the storage temperature stated" --key "passphrase"
mdcx export corpus.mdcx --target ./restored --key "passphrase"

info reads the header without the key, so the issuer and the integrity of a file can be checked before it is opened. export reconstructs the original folder, so a collection can be moved out of the format at any time.

Sent and received

Correspondence has a direction, and a question about it is usually about one side: what was asked of us, or what we answered. Where the top-level folder of a collection states that direction, it is recorded per document and a query can be restricted to it.

The names are recognised in English and Spanish, since a collection may be organised in either, and only the top-level folder is examined, so a subfolder named after a correspondent does not reclassify what it holds.

mdcx search corpus.mdcx "what was agreed about the schedule"     --key "passphrase" --only received

The MCP search tool takes the same restriction as its direction argument. A collection organised in any other way is unaffected: every document is unclassified, and a query that names no direction is not narrowed.

Working incrementally

A collection that is delivered once and a collection that grows every day place different demands on the tool. The second must not pay for what it has already done.

Conversion resumes

Conversion records the digest of each source in the Markdown it produces, and skips any file whose source is unchanged and whose output is present. This is the default; --force disables it.

The unit is the chapter rather than the document, so a book split into 47 chapters and interrupted at the 40th costs the remaining 7 on the next run. Progress is written after each unit and flushed to disk, so an interrupted run leaves a record that the next one reads.

A run over a converted collection reports what it reused:

Already converted and unchanged: 8 (reused)
Elapsed         : 0.4 min

A chapter is reconverted when its verification reported findings, since a result that was not certified is not a result worth keeping.

Packaging costs what was added

Indexing meaning dominates the cost of packaging. On one measured book: 505 seconds of encoding against 4 seconds of compression and 0.2 of encryption. Encoding the whole corpus on every publication makes adding one document cost a reindex of every previous one.

A passage whose text has not changed has the same vector. --reuse reads the vectors of an existing package and encodes only what is new:

mdcx pack --output ./Documents_md --target corpus-2.mdcx --key "passphrase" \
    --multilingual --reuse corpus-1.mdcx

  meaning indexed with BAAI/bge-m3 (1024 dimensions)
  passages encoded 395   reused 733

Measured over the chapters of one book, where 733 of 1,128 passages were unchanged, packaging took 15.4 seconds against 37.8 without reuse.

The vectors are read from the previous package, which already holds them and is already encrypted with the same key. No intermediate store is created: a vector allows the text it represents to be approximated, so keeping vectors outside the package would undo the encryption the format provides.

Reuse requires the same model. Vectors from two models occupy different spaces, so a package encoded by another model contributes nothing rather than contributing values that cannot be compared.

Several packages as one corpus

MDCX_FILE accepts more than one package, separated by the path separator of the platform or by a comma. The server queries all of them and returns one ranked list, with each result naming the package it came from.

{
  "mcpServers": {
    "mdcx": {
      "command": "python",
      "args": ["-m", "mdcx.mcp_server"],
      "env": {
        "MDCX_FILE": "/corpora/2026-01.mdcx:/corpora/2026-02.mdcx",
        "MDCX_KEY": "package-key"
      }
    }
  }
}

One key serves every package; several keys are matched to the packages in order.

This makes each package immutable: it is indexed once and never rebuilt. A corpus grows by adding packages rather than by enlarging one, which also keeps each of them within what can be decrypted into memory, since a package is decrypted whole when it is opened.

Results from different packages are merged by reciprocal rank. Their scores are computed over different corpus statistics — the frequency of a term depends on the corpus it is measured in — so the scores are not comparable between packages, while positions within each are.

MCP server

The server requires Python and this package. It does not require the conversion stack; its footprint is approximately 50 MB.

{
  "mcpServers": {
    "mdcx": {
      "command": "python",
      "args": ["-m", "mdcx.mcp_server"],
      "env": {
        "MDCX_FILE": "/path/to/corpus.mdcx",
        "MDCX_KEY": "package-key"
      }
    }
  }
}

With uv the server runs without prior installation, which is the common arrangement for Python MCP servers:

{
  "mcpServers": {
    "mdcx": {
      "command": "uvx",
      "args": ["--from", "mdcx[mcp]", "python", "-m", "mdcx.mcp_server"],
      "env": {
        "MDCX_FILE": "/path/to/corpus.mdcx",
        "MDCX_KEY": "package-key"
      }
    }
  }
}

Three tools are exposed:

The package is verified before the server begins listening, so an incorrect path or key is reported at startup rather than on the first query.

A passage carries its rank and no score. The list is ordered by that rank and by nothing else: word matching and meaning score on scales with no common meaning — one has no upper bound and depends on the corpus it was measured in, the other runs from zero to one and does not — so there is no single number here that can be compared, sorted or filtered by.

What can be compared is reported once for the reply. similarity is how near the corpus comes to the question, and a warning appears when nothing in it is about the question. The passages are returned either way: the nearest passage is worth seeing even when it is not an answer, and a corpus that answers in another language must not be hidden by this.

How near counts as near is measured from the corpus rather than fixed. Packing records answerable_at, how near this corpus comes to a question it does answer, estimated by using its own passages as questions; the reply reports it, and the warning is judged against it. A fixed threshold could not do this: the same questions reach 0.51 on one corpus and 0.55 on another, so any single cut falls inside the answered range of one collection or below another's, which is how it behaved before this was measured. A package built before this exists has no such number and is judged by the previous thresholds, unchanged.

Language support

Retrieval by word is script-aware. A query matches the words present in the documents, in any writing system, and the results of a single search may include documents in several languages. The predominant language of a corpus is recorded and reported by info; it describes the corpus and does not restrict what a query returns.

The following are verified by tests/test_languages.py, which builds one corpus holding the same four subjects — algebra, botany, printing and baking — in every language listed, then issues a query in each. Each query competes against the three other documents in its own language and against the remainder of the corpus. All 136 queries return the expected document in first position.

Support is a property of the writing system rather than of the language, so a language written in any of these scripts is covered whether or not it is listed. Three properties establish this:

Tokenisation. A word is a run of letters, digits and the combining marks attached to them. The set of combining marks is derived from the Unicode character database rather than enumerated, which keeps the vowel signs of Devanagari, Bengali, Tamil and Thai attached to the letters they modify.

Accent folding. Folding is restricted to combining marks that represent an accent placed on a letter, so that café matches cafe. The vowel signs of Indic scripts and the points of Hebrew and Arabic are preserved, since in those scripts they carry the sound of the syllable.

Segmentation. Writing systems that do not separate words with spaces — Chinese, Japanese, Korean, Thai, Lao, Khmer, Burmese, Tibetan and Javanese — are indexed by character, at index time and query time alike. This is what a lexical index can match without a segmenter trained on a single language.

Word matching operates within a language: the words of a query must be present in the document. A query written in one language reaches a document written in another only where the two share a term, as proper names and loanwords often do. When a query returns no result and none of its terms appear in the index, the response states this and names the language of the corpus, distinguishing an empty answer from material the corpus does not hold.

Retrieval across languages is a separate capability, described below. It is optional and requires a model. Where the query is written in the language of the documents the two are merged, each covering what the other cannot; where it is not, word matching has nothing to contribute and is left out, because the few terms it does match there are accidents and they arrive first.

Cross-language retrieval

Word matching operates within a language. A Spanish query and a German document on the same subject share no term, so a word index has nothing to match. Measured on a corpus written in 34 languages, a query retrieves 4.2% of the documents on its subject, comprising essentially those written in the language of the query.

Retrieving the remainder requires representing meaning rather than spelling. A multilingual embedding model places a sentence and its translation at nearby points in a vector space, so a document can be retrieved through its content rather than its vocabulary. When built with --multilingual, a package stores a vector for each passage alongside the passage, under the same encryption, and the same query retrieves 96.9% of the documents.

pip install "mdcx[multilingual]"
mdcx pack --output ./Documents_md --target corpus.mdcx \
    --key "passphrase" --multilingual

The corpus is encoded once, when the package is built. A recipient encodes only their own queries.

Merged engines

Both engines are retained because their failure modes are complementary. Measured on the same corpus of 136 documents in 34 languages:

The dense engine retrieves across languages but ranks less precisely within the language of the query. The lexical engine ranks precisely and does not retrieve beyond that language. Merging by reciprocal rank retains both properties, at a cost of one document in 136 relative to the lexical engine alone.

Ranks are merged rather than scores, as a BM25 score and a cosine similarity have no common scale.

--mode lexical and --mode semantic select a single engine.

Model selection

The default is BAAI/bge-m3. Models were compared on FLORES-200, a corpus of sentences translated by professionals into 200 languages. The task is to retrieve a sentence given its translation, among candidates drawn from the same corpus, in both directions of every language pair.

The selection criterion is the worst-performing language pair rather than the mean, since a mean can conceal a language on which a model performs poorly.

Measured over 132 language directions covering 10 writing systems, with 50 candidates per query. In a larger run of 1,122 directions across all 34 languages with 100 candidates, LaBSE reached a mean of 99.6% and 93.5% on its worst language, with no pair below 90%.

An alternative model is selected by name:

MDCX_MODEL=sentence-transformers/LaBSE mdcx pack ...

A package records the model that encoded it. A query encoded with a different model occupies a different vector space, so a mismatch disables meaning-based retrieval rather than returning results that cannot be compared.

Portable paths

No output contains absolute paths. Each document is identified by a pseudopath beginning with @/, resolved against the folder or package containing it, so a corpus remains valid on local disk, network share or cloud storage.

Signing

A package can be signed so that its issuer can be verified rather than declared. The signature covers the digest of the encrypted body, attesting to both origin and content, and is verified without the encryption key.

mdcx keygen
mdcx pack --output ./Documents_md --target corpus.mdcx --key "passphrase" \
          --issuer "Acme Ltd" --signing-key <private-key>
mdcx verify corpus.mdcx --public-key <public-key>

Verification requires the body to be intact. A signature covering only the recorded digest would accept a package whose contents had been replaced while its header was left unmodified.

The issuer field is free text and is not evidence of origin on its own.

Encryption

Packages are encrypted at rest and decrypted in memory when opened; no plaintext is written to disk. This protects a file in transit and at rest. It is not equivalent to searching over encrypted data without decryption, which is a distinct field with documented leakage attacks and per-query costs measured in seconds.

The key is derived with scrypt at N = 2^15, r = 8, p = 1. Those parameters require 32 MB of memory per attempt, which is what resists the parallelisation a GPU would otherwise bring to a search: memory, unlike arithmetic, does not become cheap by adding cores. One derivation takes 286 ms single-threaded on the development machine, approximately 3.5 attempts per second per core.

That cost falls on an attacker and on the legitimate opening of a package alike. It multiplies the work of a search; it does not make a weak passphrase safe. The strength of the encryption is the strength of the passphrase, and one drawn from a dictionary stays within reach of an offline search whatever the derivation costs.

Limitations

Retrieval returns documents whose content is close to the query. It does not translate them: passages are returned in the language in which they were written.

Documents that expose no text, such as scanned drawings, are read by optical character recognition and counted as unverifiable rather than as findings, since no text original exists against which to measure fidelity. Coverage is computed over the documents that could be measured, so an unverifiable document neither raises nor lowers it.

Coverage measures the tokens preserved by a conversion. It does not measure the preservation of table structure, which is reported separately.

Deciding that a grid of drawn rules is a table, rather than prose someone framed for emphasis, is done by how many of its rows run to more than one line. That test is not exact in either direction: prose laid out in short lines can pass it, and a table whose cells wrap can fail it. A separating signal was looked for in the material at hand and not found — the longest cell in the sample belongs to a legitimate table, so cell length does not divide them — and no further threshold was added on the strength of one collection. Where the decision goes wrong, the text is still present and still counted in coverage; what is lost is its shape.

A package is decrypted in full when it is opened, so its size is bounded by the memory available. A corpus larger than that is held as several packages and queried together, as described under Working incrementally.

Tests

pip install pytest
python -m pytest tests/ -v

254 tests. Most of them exist because something failed once; the file that covers it says which, so a correction that is undone is noticed.

Retrieval

Conversion

The package as it is installed

Tests that need a model skip themselves when it is absent, so the suite passes on a plain pip install mdcx as well as on [all]. Seven files depend on the multilingual extra and one on tables.

Continuous integration runs the suite on Python 3.11 and 3.13, on Linux and Windows.

Contributing

Issues and pull requests are accepted at github.com/jorgell23-sys/mdcx.

A change needs a test that has been seen to fail without it, and a change about cost or quality needs the measurement that justifies it. CONTRIBUTING.md sets out what makes a report act on itself and what a pull request is expected to carry; CODE_OF_CONDUCT.md covers the rest.

Security

To report a vulnerability, open a security advisory at github.com/jorgell23-sys/mdcx/security/advisories rather than a public issue.

Packages are encrypted with AES-256-GCM and keys derived with scrypt. The encryption protects a package at rest and in transit; it does not protect against a compromised host, where the key is present in memory while the package is open.

Releases

Version history and release notes: github.com/jorgell23-sys/mdcx/releases.

Versioning follows Semantic Versioning. The .mdcx format is read backwards-compatibly: a package written by an earlier version remains readable by a later one.

Authorship

Conceived and directed by Jorge Ellena G., implemented with the assistance of Claude (Anthropic).

Design decisions in this package are recorded alongside the measurements that justify them, including the ones that were rejected. Several constants here are the third value that was tried, and the two that failed are written down beside them so the next attempt starts somewhere new. Where no measurement separated two options, that is recorded too, rather than settled by a heuristic that happened to fit the material at hand.

Citation

Archived on Zenodo with a permanent identifier. The concept DOI resolves to the latest version:

https://doi.org/10.5281/zenodo.22015991

Licence

Apache 2.0. See LICENSE. Third-party components and their licences are listed in NOTICE.

PyMuPDF is not used. Its AGPL licence would require software incorporating this package to be published under AGPL, including software offered as a network service.