pubmed-proto

Generates pubmed_proto — a typed Python package for parsing NLM PubMed XML into protobuf messages — from the PubMed DTD.

This repository is the generator, not the package. It holds the inputs (pubmed.dtd, pubmed_transforms.yaml) and drives xsd-former (the xsdformer CLI) to emit the pubmed_proto source tree, which is then built into a wheel and published to PyPI. The generated tree (generated/) and build outputs (dist/) are gitignored — only the inputs are version-controlled.

Consuming pubmed_proto

Depend on the published wheel, not this repo:

pip install pubmed_proto      # or: uv add pubmed_proto

from lxml import etree
from pubmed_proto import xml_converter

tree = etree.parse("efetch_output.xml")
record_set = xml_converter.PubmedArticleSet(tree.getroot())  # XML -> protobuf

article = record_set.pubmed_article[0]  # or .pubmed_book_article
data = article.SerializeToString()      # protobuf wire bytes, for storage

An efetch response is a PubmedArticleSet of journal articles (PubmedArticle) and Bookshelf records (PubmedBookArticle — a GeneReviews chapter, say); both are modelled, with one converter per message.

The package exposes two modules (both typed; ships py.typed):

Rich-text fields

ArticleTitle, AbstractText, Keyword, CollectiveName, Citation, BookTitle, CollectionTitle and SectionTitle carry their inline markup as markdown in value: <b> becomes **…**, <i> becomes *…*, <u> becomes __…__, <sup> becomes ^(…) and <sub> becomes ~(…).

Literal characters in the text that could be read as one of those markers are backslash-escaped: \, *, _, ^ and ~ everywhere, and ) inside a ^(…) or ~(…), where an unescaped ) closes it. The markers themselves are never escaped. The title of PMID 34207366 comes out as

*Cis*-Segregation of c.1171C>T Stop Codon (p.R391\*) in *SERPINC1* Gene …

The escapes are CommonMark backslash escapes, so a CommonMark renderer shows the stop codon as p.R391*. Other markdown syntax in the text is not neutralised: a pair of backticks still forms a code span, and <…> can still form raw HTML.

For plain text, call xml_converter.markdown_to_plain_text(value), which unescapes while it drops the markers and returns Cis-Segregation of c.1171C>T Stop Codon (p.R391*) in SERPINC1 Gene …. In another language, the same thing is one left-to-right scan: a \ and the character after it stand for that character; an unescaped * or _ is dropped; ^( or ~( is dropped and opens a script; an unescaped ) is dropped if a script is open (closing the innermost one) and kept otherwise. A regex that strips the markers and ignores the escapes (re.sub(r'[*_]', '', value), say) turns the stop codon into p.R391\. Releases before 0.4.0 did not escape, so their output cannot be stripped losslessly.

Developing the generator

Requires uv.

make generate   # DTD + transforms -> generated/pubmed_proto/
make build      # generate, then build the wheel into dist/
make clean      # remove generated/ and dist/
uv run --group test pytest   # record gate over real PubMed records

Shaping the output is done in pubmed_transforms.yaml — dropping admin types, flattening list wrappers, coercing booleans/timestamps, and serializing rich-text fields to markdown. See the xsd-former docs for the transform reference.

Provenance & attribution

pubmed.dtd is derived from the U.S. National Library of Medicine PubMed DTD, version pubmed_250101 (dated 2024-08-28):

https://dtd.nlm.nih.gov/ncbi/pubmed/out/pubmed_250101.dtd

NLM DTDs are U.S. Government works and public domain in the United States; the MIT LICENSE in this repo covers CPG's own files (transforms, generator wiring, tests), not the NLM DTD.

Courtesy of the U.S. National Library of Medicine. NLM does not endorse this package. The vendored DTD and the PubMed records under tests/records/ are pinned snapshots and do not necessarily reflect the most current data available from NLM — fetch from NLM directly for current data. The records are efetch output included solely as parser fixtures. The two journal records are verbatim; their abstracts remain the copyright of their publishers or authors. Of the two Bookshelf records, pmid_25834910.xml (an NIH Molecular Libraries probe report) is verbatim: the collection's Copyright and Permissions statement reads "This publication is in the public domain", and the record carries no copyright statement. pmid_20301425.xml (a GeneReviews chapter) is trimmed: each AbstractText paragraph is replaced by a placeholder sentence and CopyrightInformation is removed, so the fixture carries only bytes we may redistribute — the gate exercises the record's structure, and its prose is not ours. Everything else in it is verbatim.

Local modification: the external MathML module include (<!ENTITY % mathml-in-pubmed SYSTEM "mathml-in-pubmed.mod"> and its reference) was removed so the DTD is self-contained for schema generation — MathML markup in titles/abstracts is not modelled. With that one include removed, the file is byte-identical to upstream pubmed_250101.

Inline markup in attribute-less text elements: Affiliation, PublisherName, Suffix and VolumeTitle are declared as text with inline markup (b, i, sub, sup, u) and nothing else — no attributes, no MathML alternative (they are exactly the %text; elements of that shape not otherwise dropped; CoiStatement is the fifth and is in drop_types). Two transforms in pubmed_transforms.yaml collapse them: drop_types removes the inline-markup types (B, I, Sub, Sup, U), which leaves each element a bare text wrapper, and inline_wrappers: true then folds it into its parent as a plain string. Both run before serialize_content, so an entry for these names has no effect, and the converter keeps only the text before the first inline element (<PublisherName>Univ of <i>Washington</i></PublisherName> parses as Univ of ). Elements with attributes or a MathML alternative stay messages and are the ones serialize_content renders to markdown.

The DTD is vendored deliberately, not fetched at build time: it's a modified derivative (so a fetch wouldn't reproduce it), and pinning the exact bytes keeps the generated schema reproducible. The full provenance also lives in a comment at the top of pubmed.dtd.

Releasing

The published version is build.version in pubmed_transforms.yaml (what xsdformer stamps into the wheel). To release:

1. Bump build.version in pubmed_transforms.yaml.
2. Publish a GitHub Release tagged vX.Y.Z matching that version.

The release workflow generates, builds, and publishes to PyPI via Trusted Publishing (OIDC). It fails if the tag and build.version disagree.