latexdetok

A tolerant LaTeX tokeniser: a .tex file read into a tree of groups, commands and text, which rewrites itself identically and never refuses valid LaTeX. Written to analyse course sources, full of personal macros and included files.

>>> from latexdetok import TexFile, expand
>>> tex = TexFile(["\\newcommand{\\R}{\\mathbb{R}}\n", "\\section{Intro}\n", "Soit $x\\in\\R$.\n"])
>>> _ = tex.analyse()
>>> [str(node) for node in tex.get_sections()[0]]
['\\section', '{Intro}']
>>> print(expand(tex).content().splitlines()[-1])
Soit $x\in\mathbb{R}$.

TeX's message, and latexdetok's

A section title that lost its closing brace, and a list closed by the wrong environment:

\documentclass{article}
\begin{document}
\section{Introduction
Energy is conserved.

\begin{itemize}
  \item $E = mc^2$
\end{enumerate}
\end{document}

pdflatex stops at the first mistake, gives no line, and names one of LaTeX's internal macros:

! File ended while scanning use of \@xdblarg.
<inserted text>
                \par
<*> intro.tex

! Emergency stop.
<*> intro.tex

latexdetok check intro.tex reports both, each where it is, with the fix:

intro.tex:3:9: error [unclosed-brace] “{” of “\section” never closed
     3 │ \section{Introduction
       │         ^
     5 │
       │ ^ first blank line of the group, line 5: the brace is probably missing before it
     9 │ \end{document}
       │ ^~~~~~~~~~~~~~ “\end{document}” comes before it is closed
       = close it with “}”

intro.tex:8:1: error [crossed-environment] “\begin{itemize}” closed by “\end{enumerate}”
     8 │ \end{enumerate}
       │ ^~~~~~~~~~~~~~~
     6 │ \begin{itemize}
       │ ^~~~~~~~~~~~~~~ opened here

2 errors, 0 warning, 0 info

What it does

• Tree: groups, environments, math, verbatim, comments, with the exact position of every element; str() rewrites the source.
• Tolerance: whatever does not match is read flat and reported (diagnostics), never refused.
• Diagnostics: check and the latexdetok check command say what is wrong, better than TeX does: the cause rather than the place where TeX gave up, the opening facing the closing, the fix, every mistake at once. Structure and meaning (\item outside a list, & outside a table, ^ outside math, a missing argument…), macros expanded, and nothing that may be valid is an error.
• Signatures: the commands of the LaTeX2e kernel bind their arguments (\section: s o m), and a hand-written table covers the common packages (amsmath, graphicx, siunitx, beamer, booktabs, enumitem…); the definitions of the file (\newcommand, \def, \NewDocumentCommand, \newenvironment…) are learned while reading.
• Catcodes: \makeatletter, \catcode, \ExplSyntaxOn, group by group.
• Inclusions: \input, \usepackage, \documentclass looked for the way TeX does (the folder, then kpsewhich), % !TEX root of chapters.
• Expanded view: the user's macros replaced by their bodies, every node tied to what produced it in the source.
• Conditionals: the ones that can be decided without compiling are, and both their branches stay in the tree.
• Typeset text: to_text renders the prose of a document, accents included, to look for a sentence in it; every character keeps its node and its position.
• Translatable messages: the diagnostics read in English or in French (set_language, LATEXDETOK_LANG); the codes, for their part, do not move.
• Edits: a modified output drawn from the analysis, the rest of the source identical byte for byte — for a minimal diff.
• Frame: an outline in text, an HTML page with two views.
• Speed: the core compiles with mypyc, twice as fast.

Installing

pip install latexdetok

The package installs the latexdetok command. For the command alone, with no Python project around it, uvx latexdetok check cours.tex runs it without installing anything, and pipx install latexdetok installs it for good.

Python 3.13 or newer, the standard library alone. To look for inclusions in the texmf trees, TeX Live (kpsewhich); without it, only the files of the document's folder are found.

A longer example

A longer course, made up for the example: twelve chapters, 96 sections, a figure and a list in each, and one mistake.

>>> import tempfile
>>> from pathlib import Path
>>> from latexdetok import Edit, TexFile, check, rewrite
>>> lines = ["\\documentclass{book}\n", "\\usepackage{graphicx}\n", "\\begin{document}\n"]
>>> for chapter in range(1, 13):
...     lines.append(f"\\chapter{{Chapter {chapter}}}\n")
...     for section in range(1, 9):
...         lines += [
...             f"\\section{{Part {chapter}.{section}}}\n",
...             f"\\includegraphics[width=6cm]{{fig/{chapter}-{section}}}\n",
...             "\\begin{itemize}\n", "\\item $E = mc^2$\n", "\\item $F = ma$\n", "\\end{itemize}\n",
...         ]
>>> lines.insert(300, "\\item a stray item\n")
>>> lines.append("\\end{document}\n")
>>> folder = Path(tempfile.mkdtemp())
>>> _ = (folder / "course.tex").write_text("".join(lines), encoding="utf-8")

Reading it, then the questions one asks of a course: its sections and where they start, its chapters cut apart, its figures, what is wrong in it.

>>> tex = TexFile(folder / "course.tex")
>>> _ = tex.analyse()
>>> len(tex.lines)
593
>>> sections = tex.get_sections()
>>> len(sections), [(found[-1].arg(), found[0].start_position[0]) for found in sections[:3]]
(96, [('Part 1.1', 5), ('Part 1.2', 11), ('Part 1.3', 17)])
>>> chapters = tex.get_lines_to_next("chapter")
>>> len(chapters), [len(chapter) for chapter in chapters[:3]]
(12, [49, 49, 49])
>>> len(tex.get_graphics()), len(tex.get_envs("itemize")), len(tex.get_envs("$"))
(96, 96, 192)
>>> for diagnostic in check(tex):
...     print(diagnostic)
301:1: error [item-outside-list] “\item” outside any list

Targeted changes: every figure to the width of the text, and the sections of the third chapter unnumbered. The rest of the file comes out byte for byte, so those lines are the only ones that differ.

>>> edits = [
...     Edit.inside(found[1], "width=0.8\\textwidth") for found in tex.get_commands_arguments("includegraphics")
... ]
>>> starts = [found[0].start_position[0] for found in tex.get_commands_arguments("chapter")]
>>> edits += [
...     Edit.of(found[0], "\\section*") for found in sections if starts[2] < found[0].start_position[0] < starts[3]
... ]
>>> edited = rewrite(tex, edits)
>>> edited.splitlines()[5]
'\\includegraphics[width=0.8\\textwidth]{fig/1-1}'
>>> pairs = zip(tex.lines, edited.splitlines(keepends=True), strict=True)
>>> changed = [number for number, (before, after) in enumerate(pairs, start=1) if before != after]
>>> len(changed), changed[:4]
(104, [6, 12, 18, 24])
>>> _ = (folder / "course-edited.tex").write_text(edited, encoding=tex.encoding)

How it compares

Written from a survey of September 2026: the code, the documentation or the issues of each project were read, and pandoc, ChkTeX and lacheck were run. The Python packages were not installed: their speed comes from their own published benchmarks. The inputs differ from one figure to the next, so only the orders of magnitude mean something.

What the others do better

• pylatexenc 3 is the serious Python alternative: MIT, no dependency, exact positions, tolerant parsing with recovery nodes. Above all, it is made to be extended from the outside — a specification brings its own parser, a construct can change the parsing state, the context database is filtered and composed by category. Describing a package latexdetok does not know means editing its data/packages.txt, that is, the package itself; and in compiled mode, latexdetok's classes cannot be subclassed. pylatexenc also converts, both ways (latex_to_text, latexencode), where to_text only renders prose to search in. Version 3 is still a beta.
• pandoc is the most complete expander among the tolerant tools: delimited \def, \edef, \let, \newif, \newenvironment, and xparse on its main branch, with years of use on every kind of document. It is not a Python library.
• ChkTeX and lacheck have decades of use behind them. lacheck runs an order of magnitude faster (57 MB/s), ChkTeX about as fast (6 MB/s), against some 3.5 million characters per second here, 7 compiled — for less work, but a quick check needs no more. ChkTeX has 49 warnings; latexdetok has 26 diagnostic codes, and TeXiFy-IDEA 75 inspections.
• texlab and TeXiFy-IDEA live in the editor, with a language server or IntelliJ's incremental analysis. latexdetok is a command, and a VS Code task that runs it.
• Overleaf's Code Check has the most readable catalogue of messages, and recovery heuristics worth copying: a precedence between delimiters, a look back at the previous \begin.
• plasTeX, LaTeXML and Texcraft are engines: real catcodes, real expansion, real .sty files executed. Where latexdetok guesses, they know — at the price of stopping on what does not run.
• digestif builds its signatures from the .sty and .ltx files actually installed. latexdetok describes about thirty packages by hand and falls back on heuristics for the rest. Reading the distribution was tried and measured: 23 times slower, thousands of names invented, kernel signatures lost.
• TexSoup is simpler, and enough for a well-formed file.

And latexdetok is young: version 0.1, one author, tried on one corpus — about 960 physics course files written by the same hand. Its rule that nothing valid is an error was checked there, on the 107 files that compile (a recent log, with no error); on other packages and other habits, expect infos and warnings it has not yet learnt to hold back. It needs Python 3.13, knows nothing of incremental analysis, and its outline and HTML page speak only English.

Where it differs

Nothing like the following was found elsewhere — which proves nothing about private tools:

1. A map from the expanded view back to the source, through nested expansions. pandoc and YaLafi give every expanded token the position of the use, a point; flachtex gives each character an origin, without the chain of expansions.
2. Conditionals decided, both branches kept. pandoc and arxiv-latex-cleaner decide, but delete the branch not taken.
3. Catcodes followed group by group, in a reader that refuses nothing. The other tolerant tools have a global flag (pandoc), one table per kind of file (digestif), re-parsed regions (unified-latex) or hard-coded values (texlab). Only the engines have the real table, and they tolerate nothing.
4. A signature deduced from the body of a macro: a \@ifstar in it makes the user's command take a star.

At a glance

When to use something else

• To convert LaTeX into text or back, or to extend the parser from your own code: pylatexenc.
• To convert a document into another format: pandoc.
• To run arbitrary .sty files faithfully: plasTeX or LaTeXML.
• For squiggles while typing: texlab, with ChkTeX.
• To query and edit sources full of personal macros, where the answer must point at the right line of the right file, and the edit must leave the rest untouched: that is what latexdetok is for.

Documentation

Diagnosing a document

python3 -m latexdetok check cours.tex
python3 -m latexdetok check chapitres/ --json

Compiler-style output, exit code 1 if there is an error.

In VS Code, two tasks turn that output into problems in the editor. Declared in the user's tasks.json rather than a workspace's, they work in every folder:

The first one keeps the terminal closed: only the Problems panel and the underlines in the editor. LATEXDETOK_LANG in the task picks the language of the messages; the problem matchers accept both, so switching it changes nothing else. The file itself is in docs/EXAMPLES.md.

Developing

uv sync --group dev
uv run pytest -q
LATEXDETOK_PURE=1 uv run pytest -q
uv run ruff check src tests scripts && uv run ruff format --check src tests scripts
uv run mypy

The core compiles with mypyc, for the interpreter that will run it; the extensions only serve if they match the sources, so one edited module is enough to fall back on them until the next build.

uv sync --group dev --group compile
uv run python scripts/compile.py

After a change to the tokeniser, on a corpus of .tex of one's own:

python3 scripts/corpus.py path/to/corpus --inputs --expand
python3 scripts/fingerprints.py path/to/corpus --inputs --expand -o before.json

Layout

The package is in src/latexdetok/; scripts/ holds the development helpers (rendering, corpus, fingerprints, bench, build, signature harvest) and tests/ the tests, including the examples of the documentation.

Licence

MIT — see LICENSE.