mdcx

Convert a document collection to verified Markdown, package it into a single encrypted file, and query it from an agent through the Model Context Protocol.

Contents

• Overview
• Requirements
• Installation
• Quick start
• Conversion
• Packaging and querying
• Sent and received
• Working incrementally
• What the corpus knows about words
• Something to keep that is not text to search
• Writing often
• MCP server
• Language support
• Cross-language retrieval
• Portable paths
• Signing
• Encryption
• Limitations
• Tests
• Contributing
• Security
• Releases
• Authorship
• Citation
• Licence

Overview

mdcx converts a collection of documents to Markdown, verifies each conversion against its original, packages the corpus with its index and provenance into a single encrypted file, and serves that file to agents over the Model Context Protocol.

It addresses one constraint. An agent asked a question about a document collection must either receive the documents in its context window, which is bounded in size and billed per token, or query a component that holds an index and returns only the passages that bear on the question. mdcx implements the second. Three properties distinguish it from an extraction script:

• Fidelity is measured, not assumed. Every conversion is checked against the text the original exposes, read by a library independent of the engine that produced the conversion, and the coverage achieved is recorded per file.
• The corpus is a single encrypted artefact. Passages, index and provenance are held in one AES-256-GCM file whose header can be read without the key.
• Every passage carries its source. An answer can be cited against a document and a location rather than recalled.

Pipeline

Conversion. Each document is attempted by the least expensive engine capable of reading it and escalated only where that engine falls short: direct text extraction, then a pass that recovers the tables a page draws, then full layout analysis. Documents exposing no text are read by optical character recognition. Content the selected engine omitted is appended verbatim rather than reported as lost.

Over the collection used during development — 99 documents, 1,144,553 reference tokens — 594 tokens were not recovered, a coverage of 99.948%. Of the 95 documents that expose text, 70 were recovered in full and none fell below 99.5%. The remaining four are scanned drawings holding no text in the file; they are marked unverifiable, as no text original exists to measure them against.

Packaging. The corpus, its search index and the provenance of every passage are written to a single .mdcx file. The development collection produced 3.9 MB from 8.8 MB of Markdown. A growing collection is not rebuilt from the start: vectors already computed are reused, and a corpus exceeding what can be decrypted into memory is held as several packages queried as one.

Retrieval. A query returns the passages that answer it, each with its source document and its position in the ranking. Word matching and dense retrieval are merged by reciprocal rank, so a query reaches a document whether it shares that document's vocabulary or only its subject, including where the two are written in different languages. Over a corpus of 136 documents in 34 languages, the merged engines rank the expected document first for 135 of the 136 queries. Where no document in the corpus is about the question, the reply states this rather than presenting its nearest passage as an answer.

Measured cost

One query over the development collection — 99 documents, 180 MB — counted with the cl100k_base tokenizer:

The 435 model tokens comprise 20 for the question, 274 for the retrieved passage and 141 for the answer.

Reading the originals costs the whole collection because a PDF is a binary format: absent prior conversion there is no way to determine which of the 99 documents holds the answer, so all of them are extracted and read.

This is a single measurement, not an average, and the saving depends on how much text an answer requires. The work is not eliminated but relocated, from the context window, which is billed and finite, to local processing, which is neither. The local column rises for that reason.

Requirements

Python 3.11 or later. No other component is required to query a package.

The floor is 3.11 because a package is held as one SQLite database and serialised in memory to be encrypted, and sqlite3 gained the call that does so in that version. Earlier interpreters were declared supported and were not: neither building a package nor opening one worked there. Conversion and cross-language retrieval each add dependencies, listed under Installation.

Installation

Querying and conversion are separated because their requirements differ by two orders of magnitude.

Conversion accounts for the heavy dependencies. A recipient who only queries an .mdcx file installs neither Docling nor PyTorch.

The multilingual extra is required for queries that cross languages. Most of its size is the embedding model, downloaded once on first use. A single-language corpus does not require it.

The tables extra covers what a page does not draw. Tables in printed material are usually found from the rules drawn around them, which costs nothing and needs no extra; borderless ones — a screenshot of a spreadsheet, a layout held together by alignment — are read by a small model that reports where the rows and columns run. It reads the shape only: the words still come from the text layer of the document, so a cell cannot hold anything the page does not say. Without it those pages are read by Docling instead, which is slower but already present in the convert extra.

If the machine has a CUDA card

Install mdcx[all-gpu] rather than mdcx[all], and install onnxruntime-gpu yourself.

onnxruntime and onnxruntime-gpu are two distributions publishing the same module, so they cannot coexist: whichever pip wrote last wins, and it is usually the CPU one. An extra that pins the CPU build therefore removes CUDA from an environment that had prepared it, on every upgrade — measured on three consecutive releases, in two environments each time, with no error, nothing in any log, and optical recognition simply costing tens of times more. all-gpu is all without that pin.

pip cannot express "either of these distributions", so this cannot be settled by declaration alone. mdcx-convert therefore checks at startup: when the machine has a card and the runtime does not offer it, it says so and gives the repair. The check on its own is one line:

python -c "import onnxruntime as o; assert 'CUDAExecutionProvider' in o.get_available_providers(), 'OCR WITHOUT THE CARD'"

Quick start

pip install "mdcx[convert]"

mdcx-convert --input ./Documents --output ./Documents_md
mdcx pack --output ./Documents_md --target corpus.mdcx --key "passphrase"
mdcx search corpus.mdcx "where is the storage temperature stated" --key "passphrase"

Conversion

mdcx-convert --input ./Documents --output ./Documents_md

The output mirrors the input directory structure, adds a global index, and records for each file the coverage achieved against its original.

Supported formats

PDF, EPUB, Word, Excel, PowerPoint, HTML, Markdown, CSV and plain text.

The format of a file is determined from its first bytes rather than from its extension. Repositories are known to serve EPUB files from URLs ending in .pdf and declaring application/pdf, where only the content identifies the format correctly. Routing such a file by extension sends it to a reader that cannot open it, and the resulting failure is indistinguishable from a damaged document.

Plain text carries no signature, so its extension determines the format. A file whose content identifies no known format is skipped rather than assumed.

How much of the machine it uses

Converting a library is the heaviest thing this package does, and it runs on a machine somebody is working at. Nothing here is a constant: every figure is derived from the machine it finds, because the same number cannot be right on four processors and on thirty-two.

mdcx-convert --input ./Documents --output ./Documents_md --max-cores 4

Processors. A fifth are left free and the rest are used: nine on twelve, three on four. The cap is a budget for the whole run rather than a grant to each process, so it is divided among the workers and each is told its share. Without that division the structured engine asks for four threads of its own and eight workers ask for thirty-two on a machine of twelve, spending inside the pool the share that was carefully left outside it.

The card. How many processes may use it at once is decided by three ceilings, the smallest winning: the free video memory divided by what a worker holds with a full batch; how much of the material actually needs a model, which is the documents that expose no text and have nothing to extract; and leaving something for the processor. All three are needed. Without the first, asking for more workers by hand does the opposite of what it looks like — twelve on a 6 GB card ask for 15.7 GB and measured three times slower than three. Without the last, a large card takes every worker and leaves one for the bulk of the work, which is processor work.

That limit is then held by a gate every worker shares, taken around the model call rather than around the document, so a process reading text is not occupying a place on the card while it does.

The batch. What a page costs falls with the number of pages it travels with — 150 ms sending one, 75 with eight, 46 with twenty-four — so the batch is as large as the card allows once every worker is seated on it, and no larger. It is decided where both halves are known, because how much of the card a worker may hold depends on how many may hold it; a worker deciding for itself reads the free memory as though nobody else would.

Lanes. Documents are dispatched to two of them. Both may reach every engine and both are counted against the same limit on the card: the lane decides what is worth dispatching where, not what a document is allowed to reach. The lane used to decide both, which meant that moving a document out of the crowded lane also took away its structured engine.

Every one of these can be overridden — --max-cores, --gpu-workers, --cpu-workers, and MDCX_TATR_BATCH — and the derived figure is the default rather than a ruling. A machine that measures differently says so.

A document the engine does not finish

There is material the structured engine does not terminate on, and it cannot be recognised beforehand: measured against documents that convert normally, the ones that hang have fewer pages, the same size, the same images per page and slightly more text. About 4% of one real collection behaved this way — two to eight ordinary A4 pages that ran for hours while their neighbours took seconds.

Without a bound, one such document holds its worker for the length of the run, and as many of them as there are workers stop the conversion altogether: the batch waits for everyone. So --timeout gives up on a document after twenty minutes by default, records it with the status TIMED OUT — its own status, not an error, because nothing was found wrong with it — and goes on to the next. --timeout 0 waits indefinitely.

The limit is deliberately generous. Abandoning a good document loses all of its work, while waiting too long for a bad one costs one worker for the excess, so it is sized for the first mistake being the expensive one.

Giving up does not undo what was started. Layout analysis says as much — the thread is likely stuck in a blocking call and will be abandoned — and an abandoned thread keeps running: measured, three of them held a core each at 100% for twenty minutes while producing nothing. A Python thread cannot be cancelled, so the only way to get the core back is to replace the process holding it, which is what --recycle-after does every fifty documents. --recycle-after 0 never replaces one.

A permit for the card is squared up per document rather than only by the block that took it, for the same reason: a block that is abandoned never returns its permit, and with two permits, two such documents left every worker waiting on a turn that never came — the card idle, the count at zero, and the run never advancing again. And if a permit were still lost beyond recovery, the run goes ahead without a turn rather than waiting forever: contention is a risk, waiting for a turn that will not come is not.

Deciding whether a book is worth converting whole

Converting a book to find out whether it is worth converting costs what the book costs. --sample-pages N converts a spread sample instead — not the first N pages, because a book opens with a cover, a blank verso and a title page, so a sample taken from the front describes the front matter rather than the book.

The sample is gathered into one document rather than converted page by page, and it keeps the headings that cutting pages would otherwise lose: a sample without them is a wall of prose, and the section titles an author wrote are most of what says whether the book is worth the rest. Its front matter says sampled: true and carries pages_total, so twenty pages of a book of six hundred cannot be mistaken for a short book.

Packing something that was never a folder

pack --output records.jsonl reads one record per line — name and text, and optionally pseudopath, folder and source — instead of walking a directory. For a collection that is generated rather than converted, writing it out as one file per document only to read it back is work with nothing to show for it: on 80,844 records, 1.4 minutes and 324 MB created, read once and deleted. A line that cannot be read is skipped and named, because one bad record should cost that record.

When a work is from

A package records a date per document and, beside it, where that date came from. Both or neither: a date without its provenance confuses when the work was published with when the file was touched, and whoever reads it cannot tell.

pack --dates dates.csv          # path,date[,provenance]
pack --date-from-mtime          # the file's time, recorded as `mtime`

The provenance is source when it came from the publisher, sidecar when somebody supplied it, front-matter when the document carried it, mtime when it is the file's time and not the work's. Where nothing reliable is found the date is absent, which is an honest answer and a different one from a guess. The copyright year printed in the text is deliberately not used: a textbook reprints its front matter, so that year is the printing's rather than the edition's.

info reports how many documents carry a date and the span they cover — 0 of 8 dated being the signal that the dates were lost on the way in. Every passage in a reply carries dated and dated_from, so the age can be shown beside the citation.

search --prefer recent orders comparable answers newest first. It enters as a third ranking fused by rank with the other two, never as a decay multiplying a score: weighting values that share no scale is precisely what fusing by rank avoids. It orders rather than filters — an older work that answers better still comes back, which matters because a work from 1970 can be the right answer, and in mathematics often is. Where both engines agree which passage is best, the date does not move it; where they disagree, it decides.

A preference can also be impossible to honour: it orders the fusion of two engines, so there has to be a fusion, and it orders by date, so something in the answer has to carry one. Neither case is an error and neither changes the answer, but from outside they look exactly like a preference that applied and found nothing to move — so the reply says which happened. The command line prints a line only when it could not be applied, and the MCP reply carries prefer_applied and prefer_reason only then; an answer without them is one where the preference ran. The reason names what to do about it, because the remedies differ: a package with no meaning index is repacked with --multilingual, one with no dates with --dates.

Where works come from

mdcx converts, packages and answers; it does not fetch, and depends on no network of its own. A catalogue is a plugin, declared through the mdcx.sources entry point group and meeting the contract in mdcx.sources. Answering questions over a package that already exists needs none of this — only building a new corpus does, and where nothing is installed it says exactly that.

Keeping the adapters out is deliberate rather than minimal. What looks like a simple HTTP client is not: one catalogue answers 403 to its whole download column and needs its handle resolved separately, another returns the same scrape cursor for every page. That knowledge belongs with whoever has it.

What does not belong there is the part that has nothing to do with any catalogue, and mdcx.sources now carries it:

python -m mdcx.sources --check <name>

checks a plugin against the contract — that search returns Candidates with identifiers it can be asked about again, that fetch returns bytes of a recognisable type or raises rather than returning something else. A plugin had nothing to check itself against, and what that cost was measured: the first thing this reports is a source returning a cover thumbnail as though it were the book. A catalogue named the attachment 9789819647453.pdf.jpg, and 5,384 bytes came back without an exception.

looks_like(data, "pdf") is the check on its own — four bytes, no network — and identify(data) says what they were instead, because this is not a PDF is not actionable and this is a JPEG is. patiently(call) retries what raises RateLimited, honouring Retry-After when the server sent one: every catalogue rate limits, and none of that is knowledge about a particular one. It catches nothing else, because a 403 on a whole download column is not transient.

tests/test_sources_kit.py holds a source in twenty lines, against no network. An executable example does not go quietly out of date.

Checking a conversion before packaging

mdcx-search searches the converted Markdown directly, before there is a package, and quotes each passage with the document and pseudopath it came from. It is how a conversion is inspected while the folder is still open to correction.

mdcx-search "movable type" --output ./Documents_md
mdcx-search --phrases ./questions.txt --output ./Documents_md --json found.json

Passages are ranked with BM25 aggregated per document rather than in isolation, so a long document that covers a subject across several fragments is not beaten by a short unrelated one that repeats a term. --literal requires the exact phrase and nothing else; --bm25 ranks by relevance without literal matching.

This engine reads the Markdown folder. Retrieval over a built package, with meaning and across languages, is mdcx search and the MCP server.

Packaging and querying

mdcx pack --output ./Documents_md --target corpus.mdcx --key "passphrase"
mdcx info corpus.mdcx
mdcx search corpus.mdcx "where is the storage temperature stated" --key "passphrase"
mdcx export corpus.mdcx --target ./restored --key "passphrase"

info reads the header without the key, so the issuer and the integrity of a file can be checked before it is opened. export reconstructs the original folder, so a collection can be moved out of the format at any time.

Sent and received

Correspondence has a direction, and a question about it is usually about one side: what was asked of us, or what we answered. Where the top-level folder of a collection states that direction, it is recorded per document and a query can be restricted to it.

The names are recognised in English and Spanish, since a collection may be organised in either, and only the top-level folder is examined, so a subfolder named after a correspondent does not reclassify what it holds.

mdcx search corpus.mdcx "what was agreed about the schedule"     --key "passphrase" --only received

The MCP search tool takes the same restriction as its direction argument. A collection organised in any other way is unaffected: every document is unclassified, and a query that names no direction is not narrowed.

Working incrementally

A collection that is delivered once and a collection that grows every day place different demands on the tool. The second must not pay for what it has already done.

Conversion resumes

Conversion records the digest of each source in the Markdown it produces, and skips any file whose source is unchanged and whose output is present. This is the default; --force disables it.

The unit is the chapter rather than the document, so a book split into 47 chapters and interrupted at the 40th costs the remaining 7 on the next run. Progress is written after each unit and flushed to disk, so an interrupted run leaves a record that the next one reads.

A run over a converted collection reports what it reused:

Already converted and unchanged: 8 (reused)
Elapsed         : 0.4 min

A chapter is reconverted when its verification reported findings, since a result that was not certified is not a result worth keeping.

Packaging costs what was added

Indexing meaning dominates the cost of packaging. On one measured book: 505 seconds of encoding against 4 seconds of compression and 0.2 of encryption. Encoding the whole corpus on every publication makes adding one document cost a reindex of every previous one.

A passage whose text has not changed has the same vector. --reuse reads the vectors of an existing package and encodes only what is new:

mdcx pack --output ./Documents_md --target corpus-2.mdcx --key "passphrase" \
    --multilingual --reuse corpus-1.mdcx

  meaning indexed with BAAI/bge-m3 (1024 dimensions)
  passages encoded 395   reused 733

Measured over the chapters of one book, where 733 of 1,128 passages were unchanged, packaging took 15.4 seconds against 37.8 without reuse.

Reuse also carries the calibration forward. A package given its questions with --focus hands them to the one built from it, so a corpus that grows keeps the threshold it was calibrated with instead of reverting to one estimated from passages — a change that barely moves the stored number and shifts the margin applied to it from 0.95 to 0.60. Passing --focus again overrides what was inherited, and the summary says when it inherited rather than doing it quietly.

The vectors are read from the previous package, which already holds them and is already encrypted with the same key. No intermediate store is created: a vector allows the text it represents to be approximated, so keeping vectors outside the package would undo the encryption the format provides.

Reuse requires the same model. Vectors from two models occupy different spaces, so a package encoded by another model contributes nothing rather than contributing values that cannot be compared.

Several packages as one corpus

MDCX_FILE accepts more than one package, separated by the path separator of the platform or by a comma. The server queries all of them and returns one ranked list, with each result naming the package it came from.

{
  "mcpServers": {
    "mdcx": {
      "command": "python",
      "args": ["-m", "mdcx.mcp_server"],
      "env": {
        "MDCX_FILE": "/corpora/2026-01.mdcx:/corpora/2026-02.mdcx",
        "MDCX_KEY": "package-key"
      }
    }
  }
}

One key serves every package; several keys are matched to the packages in order.

This makes each package immutable: it is indexed once and never rebuilt. A corpus grows by adding packages rather than by enlarging one, which also keeps each of them within what can be decrypted into memory, since a package is decrypted whole when it is opened.

Results from different packages are merged by reciprocal rank. Their scores are computed over different corpus statistics — the frequency of a term depends on the corpus it is measured in — so the scores are not comparable between packages, while positions within each are.

MCP server

The server requires Python and this package. It does not require the conversion stack; its footprint is approximately 50 MB.

{
  "mcpServers": {
    "mdcx": {
      "command": "python",
      "args": ["-m", "mdcx.mcp_server"],
      "env": {
        "MDCX_FILE": "/path/to/corpus.mdcx",
        "MDCX_KEY": "package-key"
      }
    }
  }
}

With uv the server runs without prior installation, which is the common arrangement for Python MCP servers:

{
  "mcpServers": {
    "mdcx": {
      "command": "uvx",
      "args": ["--from", "mdcx[mcp]", "python", "-m", "mdcx.mcp_server"],
      "env": {
        "MDCX_FILE": "/path/to/corpus.mdcx",
        "MDCX_KEY": "package-key"
      }
    }
  }
}

Three tools are exposed:

The package is verified before the server begins listening, so an incorrect path or key is reported at startup rather than on the first query.

A passage carries its rank and no score. The list is ordered by that rank and by nothing else: word matching and meaning score on scales with no common meaning — one has no upper bound and depends on the corpus it was measured in, the other runs from zero to one and does not — so there is no single number here that can be compared, sorted or filtered by.

What can be compared is reported once for the reply. similarity is how near the corpus comes to the question, and a warning appears when nothing in it is about the question. The passages are returned either way: the nearest passage is worth seeing even when it is not an answer, and a corpus that answers in another language must not be hidden by this.

How near counts as near is measured from the corpus rather than fixed. Packing records answerable_at, how near this corpus comes to a question it does answer, estimated by using its own passages as questions; the reply reports it, and the warning is judged against it.

That estimate is only as good as passages resembling questions, and on some collections they do not. A catalogue of 80,844 records — all back-cover blurbs, all sharing a rhetorical shape — calibrated at 0.759 where a corpus of books calibrates at 0.580, and at 0.759 the warning fires on questions the catalogue answers well. pack --focus "<question>" is for that case: given the questions a package exists to answer, the threshold is taken from them instead of estimated, and info reports which of the two it was. Repeat the option to give several; the cut goes just under the weakest of them. A fixed threshold could not do this: the same questions reach 0.51 on one corpus and 0.55 on another, so any single cut falls inside the answered range of one collection or below another's, which is how it behaved before this was measured. A package built before this exists has no such number and is judged by the previous thresholds, unchanged.

A package whose source material is gone can still be calibrated. The threshold used to be writable only by pack, and pack walks a folder of documents — so a package that outlived its material fell back to a constant nobody measured on it, and would in every future version. Nothing about the measurement needs the material: it needs the vectors, which are inside, and questions, which come from outside.

mdcx calibrate corpus.mdcx --key "passphrase"     --question "how heat travels by conduction and radiation"     --question "what the equivalence of mass and energy means"

The package is rewritten in place — same documents, same passages, same vectors, around a changed measure — and info reports it as focus-after rather than focus, because measured while packing and measured afterwards describe the corpus at different moments. A signed package needs its signing key; without it the command refuses rather than handing back an unsigned package.

The same judgement is available per package, which is what a consumer serving several of them needs in order to decide which one answers:

archive.closeness(connection, text)   # (nearest cosine, its clearance) or None
archive.answers(connection, text)     # by that package's own calibration

answers applies the margin that matches how the package was calibrated, which is the part easiest to get wrong and which raises no error when it is: applying the passage share to a threshold taken from questions was measured letting seven of eight unrelated queries through. Asking whether anything open is about a question is a different question — a property of the set — and stays where it was, in the warning the MCP server raises.

What the corpus knows about words

vocabulary(connection) returns the document frequency of every term together with the rule that produced it, and unknown_terms(connection, text) names the terms of a text this corpus has genuinely never seen.

The rule matters more than it sounds. The index records terms of three characters or more — a single character where the script does not separate words — so absence from the table has two meanings: the corpus never saw the term, or the index was never going to record it. Weighting by rarity gives an absent term the maximum weight, so reading absence as novelty makes the shortest, emptiest words the most informative ones. Measured: questions a corpus answers well declared between 0.37 and 0.55 of unknown vocabulary, and fall to exactly zero once the rule is applied. unknown_terms applies it.

unknown_terms is literal, and the index it reads is not the one that crosses languages. The meaning index reaches a Spanish question against an English corpus; the word index cannot, so asked across languages it returns every term of the text — a measure of which language the corpus is in rather than of what it knows. unfamiliar(connection, text) returns the same terms with the share and a cross_language flag. Prefer it wherever the language of the question is not known to match the corpus.

The flag is decided by the language and not by the share, which took two wrong shapes to arrive at. The share measures how much vocabulary is missing and never why: it goes high both when the corpus is in another language and when it simply does not cover the subject, and it is not even a property of the question — it rises as the package shrinks, so one query measured 0.17 against a package of 266 documents and 0.83 against one of 29, in the same language. A fixed cut on it silences small packages systematically, which are the ones for which "I have never seen these words" is the strongest thing they can say. And no cut works anyway: a real crossing was measured at 0.60 and a same-language query at 0.80.

So the detected language decides, the share only says that something is missing at all, and a detector that does not answer is not read as one that disagrees — failing to identify a language is not evidence of a different one.

Function words are not removed on top of that, deliberately. die in die casting and les in Les Misérables carry meaning in the language being searched, and which words are empty depends on the question being asked. What mdcx can state is what it never recorded; what counts as uninformative is the caller's.

The keys of df are normalised — folded case, folded accents — so a caller tokenising with search.tokenize_text gets GPU where the table holds gpu. unknown_terms handles that; anyone reading df directly must.

The two signals disagree

assess(connection, text) returns both, because each is wrong where the other is right and neither said so.

The cosine cannot tell the senses of a homonym apart — a multilingual embedding places them together. Measured on a package of algebra: graph coloring adjacent vertices different colors came in at 0.6553 against a threshold of 0.5661 and returned lessons on comparing graphs and on the ellipse. graph as the plot of a function, not as a graph. The word the question turns on, coloring, the corpus had never seen.

No quantity derived from the same vectors repairs that, which was measured rather than assumed: clearance does not separate — the false positive's falls inside the range of the questions the corpus answers and inside the range of the unrelated ones, and its closeness sits above four of eight legitimate questions — and neither does the minimum over windows of the query. Both are functions of a space that has already lost the distinction. What separates it is the literal vocabulary, which does tell an absent coloring from a present graph.

The verdict is reported, not overruled. Whether an unfamiliar word should refuse a query depends on what the query is for, and a word can be peripheral: the slope of a line drawn in Patagonia is answerable and patagonia is unknown. The MCP search reply carries unknown_terms per package — a map from the package to the words of the question it has never seen, so a reader can cross the strange word with the package the passage they are about to cite came from. Not pooled: intersecting across packages emptied the signal as the library grew, because four packages each lacked something different and no term was missing from all four, while three of them had never seen the word the question turned on. A package is absent from the map when it knows every word, and when the question is in another language than that package.

Where several packages are served, similarity is over all of them pooled and answerable_at is the lowest of their calibrated reaches, so the threshold in force comes from the narrowest package rather than from the one a passage came from. answerable_at_by_package gives each of them. The criterion is left as it is on measurement rather than preference: against the alternative — the reach of the package the best passage came from — the minimum wins six to one over 42 queries, because a question one narrow package answers would otherwise be condemned by a wider one's threshold.

Note also that vectors are stored in half precision. They are renormalised when read, because rounding to half precision costs the normalisation and every quantity computed from them was slightly not a cosine.

Something to keep that is not text to search

A corpus used as a memory sometimes has to hold an object — a certificate, a table of coordinates — and everything in the folder became passages. One such artefact of 500 vertices measured 2,003 passages, 40.6 per cent of that corpus, and made every later write cost 1.57 times as much, because packing walks the whole corpus even when one document changed.

It did not spoil the ranking, which was the fear and was wrong: coordinates resemble no question, so none of those passages reached a top five. The cost is weight and time.

indexed: false in a document's front matter keeps it in the package — signed, encrypted, one file — and out of the index, the vectors and the passage count. export restores it verbatim, and the MCP document tool returns it by name, which is the only way in, since it cannot be searched for.

pack also reports which document contributed the most passages, and says so when one holds a third or more of them.

Writing often

Compressing and encrypting are properties of the whole file, so they cost the same whether one document was added or the corpus was rebuilt. That is a fixed price per write, and it grows with the corpus rather than with what was added. pack reports it as seconds_compress and seconds_encrypt, so a caller writing often can decide how often to write.

--fast trades size for time where that is the right trade — a package that is rewritten every few minutes and never leaves the machine, rather than one that is distributed and read many times. Measured on a 30 MiB database of 190 documents: 1.03 s for 2,170 KiB against 6.22 s for 1,569 KiB. Six times the speed for 38 per cent more bytes. Nothing else about the package changes.

Language support

Retrieval by word is script-aware. A query matches the words present in the documents, in any writing system, and the results of a single search may include documents in several languages. The predominant language of a corpus is recorded and reported by info; it describes the corpus and does not restrict what a query returns.

The following are verified by tests/test_languages.py, which builds one corpus holding the same four subjects — algebra, botany, printing and baking — in every language listed, then issues a query in each. Each query competes against the three other documents in its own language and against the remainder of the corpus. All 136 queries return the expected document in first position.

Support is a property of the writing system rather than of the language, so a language written in any of these scripts is covered whether or not it is listed. Three properties establish this:

Tokenisation. A word is a run of letters, digits and the combining marks attached to them. The set of combining marks is derived from the Unicode character database rather than enumerated, which keeps the vowel signs of Devanagari, Bengali, Tamil and Thai attached to the letters they modify.

Accent folding. Folding is restricted to combining marks that represent an accent placed on a letter, so that café matches cafe. The vowel signs of Indic scripts and the points of Hebrew and Arabic are preserved, since in those scripts they carry the sound of the syllable.

Segmentation. Writing systems that do not separate words with spaces — Chinese, Japanese, Korean, Thai, Lao, Khmer, Burmese, Tibetan and Javanese — are indexed by character, at index time and query time alike. This is what a lexical index can match without a segmenter trained on a single language.

Word matching operates within a language: the words of a query must be present in the document. A query written in one language reaches a document written in another only where the two share a term, as proper names and loanwords often do. When a query returns no result and none of its terms appear in the index, the response states this and names the language of the corpus, distinguishing an empty answer from material the corpus does not hold.

Retrieval across languages is a separate capability, described below. It is optional and requires a model. Where the query is written in the language of the documents the two are merged, each covering what the other cannot; where it is not, word matching has nothing to contribute and is left out, because the few terms it does match there are accidents and they arrive first.

Cross-language retrieval

Word matching operates within a language. A Spanish query and a German document on the same subject share no term, so a word index has nothing to match. Measured on a corpus written in 34 languages, a query retrieves 4.2% of the documents on its subject, comprising essentially those written in the language of the query.

Retrieving the remainder requires representing meaning rather than spelling. A multilingual embedding model places a sentence and its translation at nearby points in a vector space, so a document can be retrieved through its content rather than its vocabulary. When built with --multilingual, a package stores a vector for each passage alongside the passage, under the same encryption, and the same query retrieves 96.9% of the documents.

pip install "mdcx[multilingual]"
mdcx pack --output ./Documents_md --target corpus.mdcx \
    --key "passphrase" --multilingual

The corpus is encoded once, when the package is built. A recipient encodes only their own queries.

Merged engines

Both engines are retained because their failure modes are complementary. Measured on the same corpus of 136 documents in 34 languages:

The dense engine retrieves across languages but ranks less precisely within the language of the query. The lexical engine ranks precisely and does not retrieve beyond that language. Merging by reciprocal rank retains both properties, at a cost of one document in 136 relative to the lexical engine alone.

Ranks are merged rather than scores, as a BM25 score and a cosine similarity have no common scale.

--mode lexical and --mode semantic select a single engine.

Model selection

The default is BAAI/bge-m3. Models were compared on FLORES-200, a corpus of sentences translated by professionals into 200 languages. The task is to retrieve a sentence given its translation, among candidates drawn from the same corpus, in both directions of every language pair.

The selection criterion is the worst-performing language pair rather than the mean, since a mean can conceal a language on which a model performs poorly.

Measured over 132 language directions covering 10 writing systems, with 50 candidates per query. In a larger run of 1,122 directions across all 34 languages with 100 candidates, LaBSE reached a mean of 99.6% and 93.5% on its worst language, with no pair below 90%.

An alternative model is selected by name:

MDCX_MODEL=sentence-transformers/LaBSE mdcx pack ...

A package records the model that encoded it. A query encoded with a different model occupies a different vector space, so a mismatch disables meaning-based retrieval rather than returning results that cannot be compared.

Portable paths

No output contains absolute paths. Each document is identified by a pseudopath beginning with @/, resolved against the folder or package containing it, so a corpus remains valid on local disk, network share or cloud storage.

Signing

A package can be signed so that its issuer can be verified rather than declared. The signature covers the digest of the encrypted body, attesting to both origin and content, and is verified without the encryption key.

mdcx keygen
mdcx pack --output ./Documents_md --target corpus.mdcx --key "passphrase" \
          --issuer "Acme Ltd" --signing-key <private-key>
mdcx verify corpus.mdcx --public-key <public-key>

Verification requires the body to be intact. A signature covering only the recorded digest would accept a package whose contents had been replaced while its header was left unmodified.

The issuer field is free text and is not evidence of origin on its own.

Encryption

Packages are encrypted at rest and decrypted in memory when opened; no plaintext is written to disk. This protects a file in transit and at rest. It is not equivalent to searching over encrypted data without decryption, which is a distinct field with documented leakage attacks and per-query costs measured in seconds.

The key is derived with scrypt at N = 2^15, r = 8, p = 1. Those parameters require 32 MB of memory per attempt, which is what resists the parallelisation a GPU would otherwise bring to a search: memory, unlike arithmetic, does not become cheap by adding cores. One derivation takes 286 ms single-threaded on the development machine, approximately 3.5 attempts per second per core.

That cost falls on an attacker and on the legitimate opening of a package alike. It multiplies the work of a search; it does not make a weak passphrase safe. The strength of the encryption is the strength of the passphrase, and one drawn from a dictionary stays within reach of an offline search whatever the derivation costs.

Limitations

Retrieval returns documents whose content is close to the query. It does not translate them: passages are returned in the language in which they were written.

Documents that expose no text, such as scanned drawings, are read by optical character recognition and counted as unverifiable rather than as findings, since no text original exists against which to measure fidelity. Coverage is computed over the documents that could be measured, so an unverifiable document neither raises nor lowers it.

Coverage measures the tokens preserved by a conversion. It does not measure the preservation of table structure, which is reported separately.

Deciding that a grid of drawn rules is a table, rather than prose someone framed for emphasis, is done by how many of its rows run to more than one line. That test is not exact in either direction: prose laid out in short lines can pass it, and a table whose cells wrap can fail it. A separating signal was looked for in the material at hand and not found — the longest cell in the sample belongs to a legitimate table, so cell length does not divide them — and no further threshold was added on the strength of one collection. Where the decision goes wrong, the text is still present and still counted in coverage; what is lost is its shape.

A package is decrypted in full when it is opened, so its size is bounded by the memory available. A corpus larger than that is held as several packages and queried together, as described under Working incrementally.

Tests

pip install pytest
python -m pytest tests/ -v

254 tests. Most of them exist because something failed once; the file that covers it says which, so a correction that is undone is noticed.

Retrieval

Conversion

The package as it is installed

Tests that need a model skip themselves when it is absent, so the suite passes on a plain pip install mdcx as well as on [all]. Seven files depend on the multilingual extra and one on tables.

Continuous integration runs the suite on Python 3.11 and 3.13, on Linux and Windows.

Contributing

Issues and pull requests are accepted at github.com/jorgell23-sys/mdcx.

A change needs a test that has been seen to fail without it, and a change about cost or quality needs the measurement that justifies it. CONTRIBUTING.md sets out what makes a report act on itself and what a pull request is expected to carry; CODE_OF_CONDUCT.md covers the rest.

Security

To report a vulnerability, open a security advisory at github.com/jorgell23-sys/mdcx/security/advisories rather than a public issue.

Packages are encrypted with AES-256-GCM and keys derived with scrypt. The encryption protects a package at rest and in transit; it does not protect against a compromised host, where the key is present in memory while the package is open.

Releases

Version history and release notes: github.com/jorgell23-sys/mdcx/releases.

Versioning follows Semantic Versioning. The .mdcx format is read backwards-compatibly: a package written by an earlier version remains readable by a later one.

Authorship

Conceived and directed by Jorge Ellena G., implemented with the assistance of Claude (Anthropic).

Design decisions in this package are recorded alongside the measurements that justify them, including the ones that were rejected. Several constants here are the third value that was tried, and the two that failed are written down beside them so the next attempt starts somewhere new. Where no measurement separated two options, that is recorded too, rather than settled by a heuristic that happened to fit the material at hand.

Citation

Archived on Zenodo with a permanent identifier. The concept DOI resolves to the latest version:

https://doi.org/10.5281/zenodo.22015991

Licence

Apache 2.0. See LICENSE. Third-party components and their licences are listed in NOTICE.

PyMuPDF is not used. Its AGPL licence would require software incorporating this package to be published under AGPL, including software offered as a network service.