likhit

likhit is Jawafdehi's public MarkItDown plugin for Nepal-specific document support.

It extends MarkItDown with Nepal-specific PDF repair, layout-aware Markdown assembly, optional OCR fallback for image-dominant PDFs, and legacy .doc support. For PDFs, likhit now evaluates multiple extraction paths and returns the best result instead of relying on a single fixed pipeline.

It also ships two things that work on transcripts after extraction: a quality audit that scores a document on independent damage axes, and personal-data detection and redaction. Both are pure-Python and importable on their own — see Transcript quality and Personal data.

Owned and maintained by Jawafdehi.

Installation

pip install likhit

Two extras name the post-extraction feature sets:

pip install "likhit[quality]"   # the transcript quality audit
pip install "likhit[privacy]"   # personal-data detection and redaction

These extras currently install nothing extra. Both feature sets are pure-stdlib, so there is no dependency for an extra to gate, and an extra cannot withhold modules from a wheel — likhit.quality and likhit.privacy are importable after a plain pip install likhit, and likhit-audit / likhit-redact are always on your PATH. The extras exist to name the separable surfaces, so a downstream requirements file can record which part of this package it depends on, and to reserve the name for the first real optional dependency.

Project Links

• Website: https://jawafdehi.org/
• GitHub: https://github.com/Jawafdehi/likhit/
• Benchmark: https://jawafdehi.github.io/likhit/ — real, published Government of Nepal documents converted with and without OCR, with the extracted Markdown and the assertions behind every result
• Contact: inquiry@jawafdehi.org

Usage

likhit is primarily used as a MarkItDown plugin.

Python

Once installed, enable plugins when creating a MarkItDown instance:

from markitdown import MarkItDown

md = MarkItDown(enable_plugins=True)
result = md.convert("path/to/nepali-document.pdf")
print(result.text_content)

MarkItDown CLI

You can also use likhit through the standard MarkItDown CLI:

markitdown --use-plugins path/to/nepali-document.pdf

To write the output to a file:

markitdown --use-plugins path/to/nepali-document.pdf -o output.md

To verify the plugin is registered:

markitdown --list-plugins

You should see likhit in the output.

likhit-save CLI

This package also installs a small helper CLI that runs MarkItDown with the likhit plugin enabled and writes Markdown files for you:

likhit-save path/to/nepali-document.pdf --out output.md

Convert multiple files into a directory:

likhit-save samples/pressrelease.pdf samples/kanunpatrika.pdf --out-dir converted/

Extract only one page or a page range from a PDF:

likhit-save path/to/nepali-document.pdf --pages 5 --out page-5.md
likhit-save path/to/nepali-document.pdf --pages 2-4 --out pages-2-4.md

Transcript quality

A whole-document Devanagari ratio is not enough to tell a good transcript from a bad one: a legacy-Kalimati report recovered by this package scored 0.977 while its first line was untranslated legacy bytes. likhit.quality scores a transcript on independent axes — legacy-ASCII leakage, mojibake, repha loss, numeric damage, structure, repetition, spacing and matra damage — and the worst axis wins, because averaging lets seven clean axes bury an eighth that is making a positive claim about damage.

from likhit.quality import audit_text

result = audit_text(markdown)
result["verdict"]            # "clean" | "suspect" | "garbled"
result["failing"]            # the axes that are not clean, worst first
result["checks"]["spacing"]  # that axis's verdict and its evidence

audit_text is pure — text in, verdicts out, no filesystem and no assumptions about how your corpus is laid out. For a directory, audit_tree walks it and takes an optional enrich callback so you can join your own metadata onto each row.

likhit-audit path/to/markdown --out report.json --workers 8

The command exits non-zero on a tree it cannot evaluate — missing, not a directory, or holding no *.md. That guard is the point: rglob raises nothing for any of those, so without it an audit of a mistyped path writes a report of zero records and exits 0, which is indistinguishable from a clean corpus.

Personal data

likhit.privacy separates finding personal data from removing it, and removes a deliberately narrower set than it finds.

from likhit.privacy import scan_text, redact_inline_text

high_precision, name_shaped = scan_text(markdown)   # counts and shapes, never a match
redacted, journal, counters = redact_inline_text(markdown)

scan_text never returns matched text, so a report of what a corpus contains can be published beside the corpus. Redaction is anchored on the label and replaces only the value, so a bare digit run can never match and a financial table cannot be touched.

likhit-redact scan path/to/markdown --out pii-report.json

# Two independent passes. Run BOTH: the first handles a label and value in one span, the
# second a value in a table cell away from its label. Either order works.
likhit-redact redact path/to/markdown --out-tree staged-1 --journal inline.json
likhit-redact redact staged-1 --out-tree staged-2 --journal tables.json --tables

--tables is not optional in practice. On this corpus, citizenship numbers held in table cells are the case the second pass exists for, and the first pass cannot see them. A recipe that omits it leaves them in place.

Exit codes are distinguishable on purpose: 0 all documents processed, 1 ran but some document failed and was skipped — the journal names each one — and 2 refused to start. A partial redaction that exited 0 would be indistinguishable from a complete one.

Redaction is never in place: it reads a tree and writes a staging copy, and refuses if the destination exists or is the source. Documents with nothing to redact are copied byte-for-byte rather than rewritten, and the journal records value shapes — lengths, digit counts, separator presence — never a matched digit.

Redaction modes

A mode names a policy tier, not a pass. One call runs whichever passes the tier needs:

from likhit.privacy import redact_text

result = redact_text(markdown)                  # redacts — this is the default
result.text, result.journal, result.values_removed

Measured over the 13 CIAA annual-report transcripts, by tier:

What is left at strict is mostly not personal data. The residual citizenship and date_of_birth counts are the scanner's wide tier matching prose: 10 of them are नागरिकता ऐन, २०६३ — the Citizenship Act and its year, a statute citation — and most of the rest are four-digit years in sentences like नागरिकता लिएकोमा मिति २०७०. The scanner is deliberately wider than the remover, and that gap is meant to be disclosed rather than chased.

The two remaining labelled-phone hits are both the Commission's own switchboard in the report masthead, where one label covers two numbers on consecutive lines and only the first is reachable — see likhit.privacy.contact for why a continuation rule is not the fix.

The 7 remaining PAN hits are meant to be there: no tier removes a PAN. See below.

The line the tiers draw: a person, not a company

🛑 No tier removes a PAN, and that is a decision rather than a gap. A PAN is a business registration number and is public by design — printed on invoices, filed with the registrar, searchable. No pattern can tell a company PAN from a sole trader's, so a rule that removed either would remove both, and removing a contractor's PAN from a procurement finding deletes the one identifier a reader can use to check the claim against the public register. PANs are detected and disclosed instead: the scanner counts them, so a corpus report still says how many are present.

That is the distinction the widest tier is built on. strict means remove the personal details — citizenship number, date of birth, phone, email, bank account, age, passport — not remove every identifier. Data about a person goes; data about a company stays, because in an accountability corpus the company identifiers are the finding.

Bilingual by requirement, not by courtesy

Every label alternation matches its English form as well as its Nepali one, and passport numbers and internationally-formatted phone numbers have rules of their own.

This is a correctness property rather than a nicety. These documents are bilingual, and the English is where the foreign parties are written: page 1 of samples/aarop-patra.pdf is Nepali prose carrying, in Latin script, one Indian national's Passport No. Z4362932, Date of Birth: 27 August 1979 and Mobile Number: 919971766006. With Nepali-only labels, strict removed the Nepali-labelled mobile two clauses earlier and left all four of those.

🛑 A Nepali-only rule does not fail at random. It fails on exactly the people whose details were written in English — which in this corpus means foreign nationals and anyone transcribed from an English-language document — so the removal set acquires a shape nobody chose. That is why this is listed as a defect rather than as coverage.

English widens what counts as a label and nothing else: every value still runs the same three guards, so a wider label cannot admit a wider value.

🛑 No tier removes names, and that is a decision rather than a gap. Nothing detects a Nepali name reliably — the only signals are श्री followed by a non-institution token (2,179 hits across those 13 transcripts) and a filled नाम: field (0). Removing parties other than the accused needs role assignment — वादी/प्रतिवादी versus witness versus relative — which is a modelling project, not a pattern. And for an accountability corpus, stripping the names of prosecuted officials would invert the product.

🛑 Redaction is the default, and unredacted output is what you ask for. likhit-save report.pdf writes a redacted transcript; likhit-save report.pdf --redact none writes the unredacted one and says so at WARNING. The reasoning is that this command's output is a file, and a file outlives the invocation that made it — nobody reading report.md six months later can tell whether redaction was considered. The safe default is the one where forgetting produces the harmless outcome.

--redact none is a legitimate thing to want — the retained original, internal casework and the research corpus all need it — so it is one flag and not a fight.

Redaction fails closed: if a pass raises, nothing is written. Falling through to the unredacted text it was asked to clean is the one outcome worse than not converting at all.

PDF in, redacted PDF out

likhit-redact pdf aarop-patra.pdf --out redacted.pdf --journal redaction.json --mode contact

This is a true removal, not a drawn box: the value is gone from the extracted text and from the file's bytes. Only the value is removed — the label beside it is kept, because redaction is located from the detector's own character offsets mapped to per-glyph rectangles rather than by searching the page for the value's digits.

🛑 It refuses any PDF it cannot read, and that is the point. Only a page whose text layer is Unicode Devanagari can be redacted today. A page carrying a legacy pre-Unicode font decodes to Latin-looking bytes, so the detectors find almost nothing in it — measured on two CIAA annual reports, the raw text layer yields 1 and 4 high-precision hits where likhit's decoded output yields 16 and 62, and in both cases the only survivors are the ASCII email addresses. "No personal data found" on such a page is not a finding, so those pages refuse the document rather than being passed through unchanged. A pure scan refuses for the same reason: it needs bounding-box OCR, which likhit's LLM-vision OCR cannot provide because it returns text without coordinates.

Document metadata, XMP and embedded attachments are scrubbed too — a document redacted only where someone looked is worse than one nobody redacted. Saves are always full saves with garbage=4, never incremental: an incremental save appends a revision and leaves the original content recoverable inside the same file.

Scope

identifiers is the formally adjudicated scope — citizenship numbers and dates of birth — and it is narrow on purpose: anchoring on the label means a bare digit run can never match, so a financial table cannot be touched.

Everything above it is a widening: the default removes the personal details rather than only the two adjudicated classes, because identifiers alone removes 0.3% of the personal data detected in the charge-sheet corpus. The one cost it knowingly accepts:

• A bank account cited as the evidence of a transfer is the claim. Removing it protects nobody and deletes the finding. strict removes it anyway; use --redact contact where the account numbers are the evidence.

PANs were the other candidate for that list and are not removed at all — see the line the tiers draw.

Passport numbers sit at strict, not at identifiers — and that is not a judgement that they are less personal. A passport number is plainly an identity-document number of the same class as a citizenship number. But identifiers is the formally adjudicated scope, and that adjudication names two classes; a caller choosing it is choosing "the scope I was authorised for", so quietly adding a third would break the only promise that tier makes.

Age is anchored on उमेर/Age and never on वर्ष or years alone. A वर्ष-adjacent rule matches 413 spans across the CIAA corpus and most of them are durations — # वर्षको लागि रु #,# is a contract term, प्रमाणपत्र जारी भएको # वर्षभित्र a deadline, सेवा प्रवेश गरेको ## वर्षमा years of service. Removing those would delete the financial evidence the corpus exists for. The उमेर anchor matches 20, of which 2 are a statutory retirement age rather than a person's.

What likhit does

likhit adds behavior beyond MarkItDown in these places:

• PDF: likhit intercepts PDF inputs, runs the default MarkItDown PDF converter first, and then decides whether to keep that result, retry with Nepal-specific extraction, or add an OCR candidate for image-dominant pages. It prefers direct likhit extraction immediately when known Nepali repair fonts are detected.
• DOC: Legacy Microsoft Word .doc files are handled by likhit's own extraction pipeline.
• DOCX: .docx files are still handled by MarkItDown's built-in Word converter, even when plugins are enabled.

Supported document types

• PDFs, including Nepal-specific born-digital PDFs and image-dominant PDFs that may need OCR
• Legacy .doc files
• .docx passthrough via MarkItDown

OCR Configuration

For image-dominant or scanned PDFs, likhit can add an OCR extraction candidate through markitdown-ocr when OCR is configured.

Required model configuration:

export MARKITDOWN_OCR_MODEL="your-model-name"

You can also provide the model through OPENAI_MODEL or GEMINI_MODEL.

Authentication options:

1. OpenAI-compatible provider with a standard OpenAI key:

export OPENAI_API_KEY="your-api-key"

2. OpenAI-compatible provider with a custom base URL:

export OPENAI_API_KEY="your-api-key"
export OPENAI_BASE_URL="https://your-provider.example/v1/"
export MARKITDOWN_OCR_MODEL="your-model-name"

3. Gemini using the OpenAI compatibility endpoint:

export GEMINI_API_KEY="your-gemini-api-key"
export GEMINI_MODEL="gemini-2.5-flash"

When GEMINI_API_KEY is set, likhit automatically uses Gemini's OpenAI-compatible base URL unless you explicitly override OPENAI_BASE_URL.

Optional variables:

export MARKITDOWN_OCR_PROMPT="Custom OCR instructions"

Recipe: a vision model on your own machine

Ollama, llama.cpp and vLLM all serve an OpenAI-compatible endpoint, so no likhit change is needed — and no page image leaves the machine:

ollama pull qwen2.5vl:7b

# Ollama ignores the key; the client refuses to start without one
export OPENAI_API_KEY="ollama"
export OPENAI_BASE_URL="http://127.0.0.1:11434/v1"
export MARKITDOWN_OCR_MODEL="qwen2.5vl:7b"

likhit-save scanned-notice.pdf --out notice.md

Recipe: AWS Bedrock, through a gateway

Bedrock does not speak the OpenAI API, so put a translating proxy in front of it. The proxy resolves AWS credentials the usual way — AWS_PROFILE, environment variables or an instance role — and likhit only ever sees an OpenAI endpoint:

pip install 'litellm[proxy]'
litellm --model bedrock/anthropic.claude-sonnet-5

# whatever you configured the proxy to accept
export OPENAI_API_KEY="local-proxy-key"
export OPENAI_BASE_URL="http://127.0.0.1:4000/v1"
export MARKITDOWN_OCR_MODEL="bedrock/anthropic.claude-sonnet-5"

Any vision-capable model your account has enabled will do; swap the model id for one you have access to.

When OCR actually runs

likhit adds an OCR candidate only for pages whose text layer cannot serve them (see Architecture below), so configuring a vision backend does not mean paying for one on every document. In the published benchmark, 13 of 16 Government of Nepal documents are converted without a single vision call; only the image-only scans need one.

If OCR is left unconfigured, likhit still converts. It logs that a page needed OCR and returns the best result it could reach without it, rather than failing:

PDF converter: OCR appears necessary, but OCR is not configured.
Set OPENAI_API_KEY or GEMINI_API_KEY, plus MARKITDOWN_OCR_MODEL, to enable markitdown-ocr.

Architecture

The high-level PDF pipeline is:

1. MarkItDown loads the plugin when enable_plugins=True or --use-plugins is used.
2. For PDF inputs, likhit reads the file and optionally slices it to the requested page range.
3. likhit scans embedded fonts. If it detects known Nepali repair fonts such as Kalimati broken-CMap fonts or legacy remap fonts, it tries the Nepal-specific extraction pipeline immediately.
4. likhit also runs the default MarkItDown PDF converter and keeps that result as a candidate.
5. likhit analyzes the PDF pages. If the file looks image-dominant with a suspicious text layer and OCR is configured, it adds an OCR candidate.
6. If the default Markdown output looks suspicious for Nepali text, likhit retries extraction with its own PDF pipeline.
7. The Nepal-specific PDF pipeline can apply:
   • Kalimati broken-CMap repair
   • Devanagari reordering
   • Devanagari spacing normalization
   • Legacy-font remapping through npttf2utf
8. After extraction, likhit checks whether the document matches a whole-document semantic structure such as a single-column notice.
9. PDF layout ordering is assigned locally while assembling content blocks, so single-column, row-aligned, and two-column regions can coexist in one file.
10. If multiple candidate outputs exist, likhit scores them and returns the best one.

Project Layout

• src/likhit/_plugin.py: MarkItDown plugin entry point and converter registration
• src/likhit/converters/: plugin converters for PDF and legacy DOC inputs
• src/likhit/nepali_pdf_repair.py: reusable Nepal-specific PDF repair layer
• src/likhit/markdown_assembly.py: generic Markdown assembly for the default conversion path
• src/likhit/extractors/: extraction strategies (PDF, DOC)
  • font_based.py: PDF extraction with Nepali font repair
  • docx_based.py: legacy DOC text extraction
• src/likhit/handlers/: structure-aware handlers and detection logic
• src/likhit/renderers/: Markdown rendering
• src/likhit/devanagari.py: Devanagari character classes and matra-damage patterns shared across the package
• src/likhit/quality/: transcript quality axes, tree walker and likhit-audit
• src/likhit/privacy/: personal-data signals, redaction passes, placeholder vocabulary and likhit-redact
• tests/: conversion, extraction, and plugin coverage
  • tests/integration/: end-to-end integration tests
  • tests/integration/test_data/: committed test fixtures (PDF, DOCX, DOC samples)

Development

likhit uses uv for dependency management, and the rest of Astral's toolchain — ruff for linting and formatting, ty for type checking.

Install the project together with its dev dependencies:

uv sync

Install the pre-commit hooks once per clone:

uv run pre-commit install

Testing

Run all tests:

uv run pytest           # ~63s
uv run pytest -n auto   # ~21s on 64 cores, ~31s on 4 — same results

-n auto (pytest-xdist) is not in addopts, because worker startup is a net loss on the single-file runs that dominate local iteration — use plain pytest for those. CI passes -n auto itself. pyproject.toml's [tool.pytest.ini_options] comment carries the measured worker-count table.

Run only the end-to-end integration suite:

uv run pytest tests/integration

Read docs/writing-tests.md before adding one. likhit reads adversarial input, and several classes of defect here are invisible to a suite that merely passes — that file covers how to prove a test bites, and the library-specific traps a test can be blind to.

The default suite is hermetic and should report executed tests only. Checks that need private PDFs, unvendored fonts, paid OCR, or an adversarial external fixture are explicit manual commands; their prerequisites and CI coverage are in docs/writing-tests.md.

Gates

CI runs the following. Run them yourself before opening a pull request:

uv run ruff check .           # lint — gated
uv run ruff format --check .  # formatting — gated
uv run pytest -n auto         # tests — gated (CI uses -n auto; plain works too)
uv run ty check               # types — ADVISORY, see pyproject.toml

ty is advisory rather than a gate: it is pre-1.0, and several of this package's dependencies (PyMuPDF among them) ship no type information, so the rule families it cannot yet see through are silenced in pyproject.toml. Read its output; do not ignore it. ty check reports a different number depending on the scope you give it and on whether the worktree has a .venv — see docs/writing-tests.md.

Before concluding a fix did or did not land in main, read Checking whether a fix is actually in main — git cherry and git patch-id are both insufficient here, in two independent ways.

References

• MarkItDown: https://github.com/microsoft/markitdown
• MarkItDown sample plugin: https://github.com/microsoft/markitdown/tree/main/packages/markitdown-sample-plugin

License

Licensed under the Hippocratic License 3.0, an Ethical Source license. See LICENSING.md for details.

Ownership

likhit is owned and maintained by Jawafdehi.

• Organization: Jawafdehi
• Website: https://jawafdehi.org/
• GitHub: https://github.com/Jawafdehi/likhit/
• Contact: inquiry@jawafdehi.org