scriptweave 2026.0.0

Parsers that turn MATLAB publish XML and jupytext percent-format Python sources into a uniform cell-based model for downstream document generation.

This package is used to parse the xml output of MATLAB publish, and this way facilitate deriving more advanced documentation that contain MATLAB code with matching output and figures.

A sibling submodule, scriptweave.jupyter, provides the same extraction model for Python notebooks expressed in the jupytext percent format. Unlike the MATLAB flow, the Python parser executes the source as part of parse() using a Jupyter kernel and captures stdout, image outputs, and rich results.

Both backends now map into a shared normalized representation defined in scriptweave.model:

• ParsedDocument for file-level metadata and ordered cells
• Cell for per-cell content, text rendering, outputs, and images

Backend-specific values are surfaced through shared metadata fields on the document (matlab_release, matlab_version, kernel_name, python_version) and through backend / execution_mode.

Migration notes (old -> new assumptions)

If you consumed parser results before the shared model change, these are the relevant adjustments:

• scriptweave.matlab.parse(...) and scriptweave.jupyter.parse(...) still exist and remain the primary entry points.
• Returned objects now share the same base shape (ParsedDocument and Cell) even though backend-specific names (MatlabFile, PythonFile) are still exported for compatibility.
• Prefer capability-style checks over backend class checks:
  • use doc.backend and doc.execution_mode
  • instead of isinstance(doc, MatlabFile) / isinstance(doc, PythonFile)
• Backend metadata moved to common accessors:
  • MATLAB: doc.matlab_release, doc.matlab_version
  • Jupyter: doc.kernel_name, doc.python_version
  • all are also available in doc.metadata
• Cell payload access is aligned:
  • use cell.source for canonical cell input text
  • cell.code is still available for code cells and remains a compatibility alias
  • use cell.text_as_str() and cell.text_as_latex() for rendered text across both backends

In short: keep your existing parse entry points, but treat outputs as one normalized document format with backend metadata attached.

Usage

MATLAB processing

Write the code that you want to create output from as a MATLAB publish document. The package will parse the code, and separate the different cells. The package will separate documentation, executed code, console output and figures.

In MATLAB process the created file (for example cubic_plot_publish.m) using publish:

publish('cubic_plot_publish.m',...
        'format', 'xml',...
        'imageFormat', 'epsc',...
        'outputDir', 'gen/',...
        'catchError', false);  % Fail on thrown exception

This will produce the file gen/cubic_plot_publish.xml

Python Processing

Usage:

from pathlib import Path

from scriptweave.matlab import parse

mf = parse(Path('cubic_plot_publish.xml'))

print(mf.filename)  # The name of the processed file
print(mf.output_dir)  # Directory for generated files
for c in mf.cells:
  print(c.title)  # Title of cell
  print(c.code)  # Executed code in cell
  print(c.output)  # Console output from cell
  ...

print(mf.backend)  # "matlab"
print(mf.execution_mode)  # "artifact"

Explore the object to figure out what has been extracted.

Current fixture examples in tests/data/ are: cubic_plot_publish.m, formatted_text_publish.m, text_only_publish.m, single_figure_publish.m, and multi_figure_publish.m.

HTML → LaTeX conversion

Cell.text_as_latex() converts the small HTML fragment that MATLAB publish emits inside each <text> element into LaTeX. The conversion is performed by a tiny in-tree module, scriptweave.html_to_latex, rather than a third-party library.

Rationale. Earlier versions depended on the unmaintained html2latex package, installed straight from a personal GitHub fork. The package is not on PyPI, has not tracked changes in its own dependencies (notably justhtml), and pulls in a heavy HTML-parsing stack to handle constructs MATLAB publish never produces. General HTML-to-LaTeX libraries (e.g. routing through Pandoc via pypandoc) were considered, but they add either a non-Python runtime dependency or substantially more surface area than this project needs.

The MATLAB-publish HTML subset is narrow and stable — paragraphs, basic inline emphasis (<b>, <i>, <tt>), simple lists, links, line breaks, preformatted/verbatim blocks, inline and display LaTeX equations, external images, raw LaTeX blocks, and (dropped) raw HTML blocks — so a small purpose-built converter is both easier to test and easier to keep working. The set of tags handled was derived directly from real MATLAB publish('...','format','xml') output (see tests/data/all_markup_publish.m).

MATLAB-specific notes.

• <equation> is emitted by publish with the original LaTeX source preserved on either the <equation text="$$...$$"> attribute (display equations) or on the inner <img alt="$...$"> (inline equations). The converter prefers those over re-deriving math from rendered pixels.
• <latex> blocks contain raw LaTeX (e.g. \begin{tabular}...) and are emitted verbatim, not wrapped in math delimiters.
• <html> blocks have no faithful LaTeX equivalent and are dropped with a UserWarning.
• <mcode-xmlized> (sample code with a three-space comment indent) is rendered as a \begin{verbatim}...\end{verbatim} block, stripping the namespaced mwsh:* syntax-highlight tags.
• MATLAB emits the trademark symbols ® / ™ as literal Unicode, which passes through the converter unchanged.

Extending the converter. The converter dispatches each HTML tag through a registry of handlers. To support a new tag without forking the module, register a handler on the default renderer:

from scriptweave.html_to_latex import HtmlToLatex, register_tag

def _underline(element, renderer: HtmlToLatex) -> str:
    return f"\\underline{{{renderer.render_element(element)}}}"

register_tag("u", _underline)

A handler receives the xml.etree.ElementTree.Element it is rendering and the active renderer (so it can recurse via renderer.render_element / renderer.render_children). For private rendering pipelines that should not mutate the module-level default, instantiate HtmlToLatex() directly and call its register / convert methods.

Unknown tags do not raise: they render their children transparently and emit a UserWarning. This keeps existing pipelines working when MATLAB introduces new constructs, while still surfacing them so a handler can be added.

Jupyter / jupytext processing

For Python sources, write your script in jupytext percent format. A cell begins with # %%; markdown cells use # %% [markdown] and prefix each content line with # . Optional cell titles follow the # %% marker:

# %% [markdown]
# # Cubic plot example
#
# A short description of the experiment.

# %% cubic_plot
%matplotlib inline
import matplotlib.pyplot as plt

xs = list(range(11))
ys = [x ** 3 for x in xs]
print(f"x_max={xs[-1]} y_max={ys[-1]}")

fig, ax = plt.subplots()
ax.plot(xs, ys)
plt.show()

Parse and execute the file:

from pathlib import Path

from scriptweave.jupyter import parse

pf = parse(Path("cubic_plot_publish.py"))

print(pf.filename)        # Source file name
print(pf.output_dir)      # Where extracted images were written (defaults to ./gen)
print(pf.kernel_name)     # "python3"
print(pf.python_version)  # Reported by the kernel
print(pf.backend)         # "jupyter"
print(pf.execution_mode)  # "executed"

for cell in pf.cells:
    print(cell.id, cell.cell_type, cell.title)
    if cell.cell_type == "markdown":
        print(cell.text)              # Raw markdown source
        print(cell.text_as_str())     # Rendered HTML
        print(cell.text_as_latex())   # Rendered LaTeX (via scriptweave.html_to_latex)
    else:
        print(cell.code)              # Source code
        print(cell.output)            # Captured stdout / stream text
        print(cell.results)           # text/plain values from execute_result
        print(cell.images)            # Saved PNG / SVG file names in output_dir

parse() accepts an explicit output_dir, a kernel_name (defaults to python3), a per-cell timeout, and an execution_policy:

• execution_policy="execute" (default): execute code cells and raise ParseError on execution failure.
• execution_policy="skip": parse structure only; code cells keep source but have no runtime outputs/results/images.
• execution_policy="best_effort": execute all code cells, continuing past failing cells. Per-cell failures are recorded on each cell's error field; kernel-level failures (e.g. startup or timeout) are stored in pf.metadata["execution_error"] while preserving any partial outputs. If anything failed, execution_mode is "best_effort", otherwise "executed".

For non-skip execution policies, parser metadata also includes pf.metadata["execution_diagnostics"] with summary fields: total code cells, code cells with outputs, code cells with errors, and the first failed cell index (or None).

The parser suppresses kernel-process stderr during startup so benign IPKernelApp warnings from the local Jupyter runtime do not leak into caller output.

Logging

The library uses Python's standard logging module and emits structured messages from both parser backends:

• scriptweave.matlab.parser: XML parse lifecycle and parse failures.
• scriptweave.jupyter.parser: source parsing, execution-policy decisions, notebook execution outcomes, and parse summaries.

By default, the package installs a NullHandler, so importing it does not configure global logging or emit messages unless your application enables them.

Example:

import logging

from scriptweave.jupyter import parse

logging.basicConfig(level=logging.INFO)

document = parse("example_publish.py")

Set the level to DEBUG to see detailed execution diagnostics.

Recommended logger names and levels:

• scriptweave: set to INFO for high-level lifecycle messages.
• scriptweave.matlab.parser: set to DEBUG to troubleshoot MATLAB XML parsing.
• scriptweave.jupyter.parser: set to DEBUG to troubleshoot notebook execution and policy handling.

Example with per-logger levels:

import logging

logging.basicConfig(level=logging.WARNING)

logging.getLogger("scriptweave").setLevel(logging.INFO)
logging.getLogger("scriptweave.jupyter.parser").setLevel(logging.DEBUG)

# logging.getLogger("scriptweave.matlab.parser").setLevel(logging.DEBUG)

Current Jupyter fixture examples in tests/data_jupyter/ are: cubic_plot_publish.py, formatted_text_publish.py, and text_only_publish.py.

Testing

Run the test suite with:

pytest

Optionally run tests with coverage reporting:

pytest --cov=scriptweave --cov-branch --cov-report=term-missing

When you change a MATLAB source fixture in tests/data/*.m, regenerate the matching published XML fixtures in tests/data/gen/*.xml by running tests/data/runme.m in MATLAB before running the tests again. The suite checks both that every source fixture has a matching XML file and that the XML originalCode still matches the current MATLAB source.

Output rendering expectations are stored as snapshots in tests/snapshots/. This includes per-cell rendering snapshots and full parsed-file snapshots for all published examples. When output formatting intentionally changes, refresh snapshots with:

pytest --update-output-snapshots

When adding a new example fixture:

1. Add the `.m` file to `tests/data/`
2. Add it to `tests/data/runme.m`
3. Generate the matching XML in `tests/data/gen/`
4. Run `pytest --update-output-snapshots` if the parsed output changed intentionally

When adding a new Jupyter fixture:

1. Add the `.py` percent-format file to `tests/data_jupyter/`
2. Run `pytest --update-output-snapshots` to capture the executed snapshot

The Jupyter tests require a working python3 Jupyter kernel (ipykernel) and the runtime dependencies nbclient, nbformat, and markdown. matplotlib is pulled in as a dev dependency so that the figure-producing fixtures can execute end-to-end.

License

This project is licensed under the MIT License. See LICENSE.md for the full text.