markstay , Python reference implementation (v1 core)

The Python reference implementation of the markstay spec (v1.5). markstay is a source-level identity primitive for Markdown blocks: an id token that stays bound to its block across edits (marker stay:), so a reference to a block survives the document being rewritten, including by an LLM.

This is the parser-free core: everything string-level and parser-independent (§8 hashing, §3/§4 marker grammar, §5 blank-line segmentation, §6 id minting, the §3/§4/§7/§8 write path, §7/§11 lint, §9 quote recovery, §9.1 resolution ladder). It mirrors the JavaScript reference (markstay on npm); both are gated by a shared language-neutral conformance corpus, which turns "two implementations agree" from an assertion into a tested fact.

Child-block identity (§5.5) is implemented here, behind --child-blocks on the CLI and child_blocks=True in the API: a direct list item may carry its own stay under the reserved subhash key, resolved through the §9.2 ladder. It is opt-in because §16 makes segmenting and resolving child blocks optional. The write-path shim §16 does make mandatory is unconditional here, as it is in every implementation.

Install

pip install markstay

Zero runtime dependencies (Python standard library only). CommonMark-tree segmentation (§5.2) is an optional extra:

pip install "markstay[commonmark]"   # pulls in markdown-it-py

Requires Python >= 3.9.

Keeping stays alive through an agent's edit (start here)

Almost every stay that goes missing goes missing the same way: a model rewrote the document and did not know the markers were load-bearing. The eval measured both halves of the fix, and they are not close , a naive "clean this up" rewrite keeps about 5% of markers, the same rewrite carrying the SPEC.md §11 instruction keeps ~96-100%, across five models and three vendors. That outweighs model tier.

markstay preserve                        # the §11 instruction, ready to paste into
                                         #   AGENTS.md / CLAUDE.md / a system prompt
markstay preserve --wrap DOC.md          # the instruction wrapped around a document,
                                         #   as a complete editing prompt
markstay preserve --wrap DOC.md --task "Rewrite this to be clearer."

import markstay as M

M.PRESERVE_INSTRUCTION            # the §11 contract, worded for an editing agent
M.preserve_wrap(doc, "Tighten it.")   # the prompt shape the eval measured

The instruction is byte-identical in the npm and crates.io packages, held there by the shared conformance corpus rather than by convention.

Everything below is the backstop: it catches loss after the fact, it does not prevent it. Ship the instruction first.

Library

import markstay as M

md = "The ingest stage retries three times.\n<!-- stay:a1b2 -->\n"

# parse into content blocks with attached markers (§5)
blocks = M.parse_document(md)

# well-formedness + intra-doc invariants (§7): duplicate/orphan/malformed/drift
_, findings = M.lint_document(md)

# regeneration diff (§11): what an edit did to the ids (dropped/duplicated/moved)
findings = M.lint_diff(before_md, after_md)

# §8 content hash (ASCII-normalized SHA-256)
M.body_hash("some block body")

# §9.1 resolution ladder: re-attach ids after an edit, or report DETACHED
anchors = M.build_anchors(before_md)
resolutions = M.resolve(anchors, after_md)   # id -> marker | hash | quote | detached

detached = resolutions["a1b2"]
detached.reason       # ambiguous | unmatched, or a child-path reason
detached.candidates   # diagnostic contenders for ambiguity, never a committed target

# write path: mint ids for unmarked blocks (§6), append the §3.1 trailing marker
res = M.stamp("First paragraph.\n\nSecond paragraph.\n")
res.text     # each block now carries <!-- stay:ID hash=sha256:... -->
res.minted   # [{"id": ..., "line": ...}, ...]

# refresh a hash you edited on purpose (§8); repair duplicate ids (§7, copy mints new)
M.restamp(edited_md)            # -> RestampResult(text, refreshed)
M.repair_duplicates(copied_md)  # -> RepairResult(text, renamed)

Experimental list-item identity

The canonical Python reference includes an opt-in list-item prototype. It is not part of the spec and has no cross-language parity, so it is not a portable format promise. Enable it explicitly:

seeded = M.stamp(md, mode="commonmark", child_blocks=True).text
_, findings = M.lint_document(seeded, mode="commonmark", child_blocks=True)
findings = M.lint_diff(before, after, mode="commonmark", child_blocks=True)

anchors = M.build_child_anchors(seeded, mode="commonmark")
resolved = M.resolve_children(anchors, after, mode="commonmark")

Child markers are inline and use subhash=sha256:...; the same stamping pass mints the containing list's parent stay. restamp(..., child_blocks=True) refreshes subhash and never injects a parent hash into a child. If an older restamp --add-missing already added that parent hash, repair_duplicates(..., child_blocks=True) removes the detectable residue.

CommonMark mode handles direct list-item source spans. The dependency-free mode fails closed outside flat, tight, single-paragraph lists. Measured attachment safety is 0% false attachment from two independent directions (0/324 deterministic, 95% upper bound 0.92%; 0/256 on real gpt4o rewrites, bound 1.16%) at 95.0% and 92.2% recovery. The feature stays experimental because its benefit is unmeasured, not because its safety is in doubt.

Public API (the spec'd portion mirrors the JS index.js surface; child names are experimental Python-only): normalize_body, body_hash, Marker, find_markers, strip_markers, strip_markers_outside_code, rewrite_markers, code_lines, fence_state, segment_blank_line, segment_commonmark, segment_child_items, Block, ChildBlock, child_body, parse_document, Finding, lint_document, lint_diff, sort_findings, has_errors, mint_id, ID_CHARSET, format_marker, format_attr_value, stamp, restamp, repair_duplicates, DEFAULT_HASH_LENGTH, Selector, normalize, body_score, context_bonus, rank_candidates, best_match, CONTEXT_CHARS, Evidence, Candidate, Anchor, Resolution, build_anchors, resolve, ChildAnchor, ChildResolution, build_child_anchors, resolve_children, DEFAULT_THRESHOLD, DEFAULT_MARGIN, PRESERVE_INSTRUCTION, PRESERVE_RETURN_ONLY, preserve_wrap, CommitEntry, StagedCheck, check_entries.

CLI

markstay preserve                     # the §11 instruction for an editing agent
markstay preserve --wrap DOC.md       # that instruction + the doc, as a prompt
markstay lint    FILE [FILE ...]      # well-formedness + intra-doc checks
markstay lint    --before OLD.md NEW  # regeneration diff (dropped/duplicated/relocated ids)
markstay lint    --json ...           # machine-readable findings
markstay lint    --commonmark ...     # §5.2 CommonMark-tree segmentation (needs the extra)
markstay lint    --child-blocks --commonmark ...  # experimental list-item identity
markstay resolve --before OLD.md NEW.md  # explain each attachment or detachment
markstay resolve --before OLD.md NEW.md --show-candidates  # show ambiguous contenders
markstay resolve --before OLD.md NEW.md --json  # versioned structured diagnostics
markstay check-staged [FILE...]       # the same diff against the staged commit
markstay check-worktree [FILE...]     # check files on disk before the next commit
markstay stamp   FILE... [-w]         # mint ids for unmarked blocks (§6)
markstay restamp FILE... [-w]         # refresh hashes that drifted (§8)
markstay repair  FILE... [-w]         # mint fresh ids for duplicate ids (§7)

--child-blocks is also accepted by check-staged, check-worktree, stamp, restamp, and repair. It remains off unless requested.

resolve always requires the marked baseline and the edited document. Its normal text output reports every state but keeps candidate details behind --show-candidates. JSON uses markstay.resolve/v1; detached entries carry committed: false and a markstay.resolve-diagnostics/v1 object containing the failed threshold and margin context, structured evidence codes, optional human labels, and candidate provenance. The candidate set is diagnostic and is not an attachment recommendation. unmatched omits sub-threshold candidates by design.

Block detachments use ambiguous or unmatched. The child path can also report unscored when no match was attempted, unaddressed for a surviving child marker that owns no direct child, and contested when multiple stays propose the same tier-start candidate. A contest carries the proposed target and the ids of the other stays that proposed it, while keeping candidates empty. Snapshot context evidence is labelled as adjacency within that filtered candidate snapshot. A child contest still continues to weaker tiers as §9.2 requires; only a stay that remains detached reports its strongest contest. A child blocked by an unresolved parent carries that parent resolution in blocked_by; a surviving child marker still resolves independently.

lint exits non-zero when any error-level finding is reported, so it gates a commit hook or an agent's post-edit step. The write verbs print the result to stdout by default; -w/--write edits files in place.

Gating commits (pre-commit framework)

lint needs two files. check-staged needs only a repo: it reads the staged commit and finds each document's baseline itself, which is what a hook actually wants.

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/markstaymd/markstay-py
    rev: v0.9.0
    hooks:
      - id: markstay                  # or markstay-collections, to include table
                                      # rows and list bullets

It stays quiet unless there is something to act on: a commit that only edits stamped blocks in place or mints new ids prints nothing, so the channel keeps meaning something. --show-drift opts back in, --json for machine output.

The baseline is resolved by stay id, not by filename. git's rename detection is content-similarity based, and similarity is anti-correlated with this failure mode: the more a rewrite destroys, the more stays it can drop and the less git sees a rename. A measured real case scored 2% similarity, so git recorded delete + create and a path-keyed baseline found nothing to compare against. A surviving stay id is the stronger signal. An id that moved to another document in the same commit is reported as a move rather than a loss, so reorganising documents does not block.

Segmentation notes

• Leading YAML frontmatter is metadata, not a block (§5.3): it is skipped before segmentation, so it is never a block, never stamped, and never hashed , a metadata-only edit (status: draft -> status: done) must not read as a content edit. Recognition is conservative, because --- is also a thematic break and a setext underline: a span counts only when line 1 is exactly ---, a later line is exactly --- or ..., the payload between them is non-empty with no blank line, and at least one payload line is unambiguously YAML (a key: or a - item). A YAML comment does not count, since # x is also an ATX heading. "Unambiguously YAML" is judged with ASCII whitespace, as everywhere else in the spec (§8/§9): the runtimes' own Unicode whitespace sets disagree with each other, and a rule that DELETES a span must not vary by implementation. The conditions confine the ambiguity rather than removing it, and the rule does not pretend otherwise: a blank-free payload that reads as YAML is also ordinary Markdown, whether it is a sequence (--- / - Keep this / ---, a list between two thematic breaks) or a mapping (--- / title: v / ---, a setext heading under one). Both are accepted and their content is excluded. Frontmatter wins, the same call every mainstream Markdown site generator makes on the same bytes. A document that fails any of the four conditions (no opening ---, no closing fence, a blank line in the payload, no payload line that reads as YAML) falls through to ordinary Markdown, where the worst case is a spurious block and a stray hash-drift warning. A marker an older version stamped onto frontmatter usually raises ORPHAN_MARKER, and deleting that one marker is the whole migration. Lint before deleting: with no blank line between the marker and the content below it, blank-line segmentation reads one run and binds the marker to that content, so it is live rather than orphaned and deleting it drops a working id.

The conformance corpus (the actual deliverable)

The corpus under conformance/ is shared with the JavaScript reference. 408 vectors across two tiers. The check category supplies 13 commit-shaped cases with paths, statuses, before/after text, expected baseline pairings, findings, move/deletion/tracking-departure notes, and scope behavior.

• spec/ , hand-authored from the spec prose, asserting what the words require. These are authority; a spec/ vector the reference fails is a reference bug, not a corpus error.
• gen/ , emitted from the reference for breadth/regression.

The JS reference runs the same JSON, so the two runners are a cross-impl regression sentinel: any later change to either implementation that breaks agreement fails one of them.

Running the tests

pip install -e ".[commonmark]"
pytest

License

MIT