flexdoc 0.3.0

Document model for powerful text parsing and processing in Markdown

FlexDoc

A Markdown parser gives you a block AST but no sentences, sizes, or exact source offsets. An NLP toolkit gives you sentences but no Markdown structure. If you need to know which sentence is in which section, how many words (or LLM tokens) a section holds, or exactly where a link sits in the original text, you end up gluing tools together; the glue breaks on the first edit.

FlexDoc builds a single source-grounded model and exposes its structure as layers over one shared coordinate space: exact [start, end) offsets into one retained, normalized source string. The Markdown layer (blocks, inline elements, typed attributes), the textual layer (paragraphs, sentences, word tokens), and the document layer (heading hierarchy, table of contents) are independent parses of the same text, so cross-cutting questions are simple offset queries. One query primitive (collect()) spans all layers; DocGraph serializes any slice as language-neutral JSON; SpanRef anchors spans by quoted text so references survive edits and reparses.

FlexDoc is a standalone library. chopdiff builds its diff-filtering and windowed-transform layer on top of flexdoc.

Installation

uv add flexdoc
# or: pip install flexdoc

Status

Beta (0.3.x). The core model is established, but the later-stage mechanisms in the stabilization roadmap remain open. Breaking changes before 1.0 bump the minor version, so pin a minor version. See the changelog.

Usage

The primary entry point is FlexDoc:

from flexdoc import FlexDoc, NodeKind, TextUnit

markdown_text = "# Introduction\n\nSee [docs](https://example.com/docs).\n"
doc = FlexDoc.from_text(markdown_text)

# Section hierarchy with rolled-up sizes:
print(doc.section_size_tree(units=(TextUnit.words, TextUnit.sentences)))
# # Introduction  (4 words, 2 sentences)

# Sizes at every grain, including approximate LLM tokens:
print(doc.size_summary())
# 53 bytes (3 lines, 2 paras, 2 sents, 4 words, ~14 tok)

# One query primitive across all layers:
link = doc.collect(kinds={NodeKind.link})[0]
print(link.attrs["url"], link.source_span)
# https://example.com/docs (20, 52)

# Round-trips back to normalized Markdown:
print(doc.reassemble())

Every located source-backed unit carries an exact [start, end) span into normalized source_text; paragraphs and sentences also expose that slice as original_text. Some parser-derived identities can be unlocatable and therefore carry span=None.

FlexDoc delegates Markdown parsing to marko (CommonMark with GFM tables and footnotes) via flowmark, and adds sentence segmentation, the section hierarchy, the flat node table, offset-grounded queries, serialization, and span anchoring on top. Parsing any input never raises: malformed Markdown degrades deterministically and visibly (see the spec and the golden-test corpus that pins this behavior).

The full public surfaces live in the submodules:

• flexdoc.docs: FlexDoc, Paragraph, Sentence, Section, Block, BlockType, the node table, collect(), DocGraph, SpanRef, and source-linked render/report helpers.
• flexdoc.docs.wordtoks, flexdoc.docs.search_tokens, flexdoc.docs.token_diffs, and flexdoc.docs.token_mapping: lower-level token, search, diff, and mapping utilities that are not promoted by flexdoc.docs.
• flexdoc.html: html-in-md, html/plaintext conversion, HTML tag helpers, the content extractor, and timestamp extraction.
• flexdoc.util: read-time and token-count estimation.

See usage.md for the main workflows, and the worked examples (run with uv run python examples/<name>.py from a checkout):

• doc_structure.py covers section hierarchy, size rollups, offset lookups, and the block tree.
• normalized_form.py covers block-type tallies, list-density invariance, and per-section links.
• backfill_timestamps.py aligns an edited transcript to its timestamped source via token mapping.

Project Docs

For users:

• Design of record (the full spec): flexdoc-spec.md.
• Installing uv and Python: installation.md.

For contributors:

• Development workflows: development.md.
• Publishing to PyPI: publishing.md.
• Dependency policy: SUPPLY-CHAIN-SECURITY.md.
• Design history and plans: docs/project/specs/active/.

---

This project was built from simple-modern-uv.