mdcmd 0.7.5

Execute commands in Markdown files, embed output, generate TOCs

mdcmd

Execute commands in Markdown files, embed output, generate TOCs

e.g.:

seq 3
# 1
# 2
# 3

☝️ This block is updated programmatically by mdcmd (and verified in CI; see raw README.md).

• Overview
• Install
• mdcmd: execute commands in Markdown files, embed output
  • bmdf example
  • HTML example
• bmd: format bash command and output as Markdown
  • bmdf (bmd -f): command+output mode
  • bmdff (bmd -ff): two-fence mode
  • bmdfff (bmd -fff): <details> mode
  • bmdc (bmd -y console): <console> mode
  • Piping
  • Env vars
  • -w/--workdir / $BMDF_WORKDIR
• toc: Markdown Table of Contents
• Examples
• GitHub Action

☝️ This TOC is generated programmatically by mdcmd and toc (and verified in CI; see raw README.md).

Overview

This package provides 3 CLIs:

• mdcmd: execute shell commands in Markdown files, embed output
• bmd: run Bash commands, wrap output for Markdown embedding
  • Useful in conjunction with mdcmd
  • bmdf, bmdff, bmdfff, bmdc provide different output styles (or use bmd -y/--style)
• toc: generate Markdown table of contents (with custom "id"s for sections)
• mktoc: convenience wrapper for mdcmd -x '^toc$'

Install

Global install via pipx or uv (recommended):

pipx install mdcmd
# or: uv tool install mdcmd

You can also install in the current (v)env:

pip install mdcmd

mdcmd: execute commands in Markdown files, embed output

mdcmd --help

Usage: mdcmd [OPTIONS] [PATH] [OUT_PATH]

  Parse a Markdown file, updating blocks preceded by <!-- `[cmd...]` -->
  delimiters.

  If no paths are provided, will look for a README.md, and operate "in-place"
  (same as ``mdcmd -i README.md``).

Options:
  -a, --amend                     Squash changes onto the previous Git commit;
                                  suitable for use with `git rebase -x`
  -C, --no-concurrent             Run commands in sequence (by default, they
                                  are run concurrently)
  -i, --inplace / -I, --no-inplace
                                  Edit the file in-place
  -n, --dry-run                   Print the commands that would be run, but
                                  don't execute them
  -T, --no-cwd-tmpdir             In in-place mode, use a system temporary-
                                  directory (instead of the current workdir,
                                  which is the default)
  -x, --execute TEXT              Only execute commands that match these
                                  regular expressions
  -X, --exclude TEXT              Only execute commands that don't match these
                                  regular expressions
  --help                          Show this message and exit.

# Modify README.md in-place
mdcmd -i README.md
# Same as above; no args defaults to `-i README.md`
mdcmd

That's how the various command examples in this file are generated / updated!

bmdf example

The example at the top of this file is generated by a line like:

<!-- `bmdf seq 3` -->

mdcmd transforms that into:

<!-- `bmdf seq 3` -->
```bash
seq 3
# 1
# 2
# 3
```

Notes:

• HTML comments (<!-- ... -->) are hidden in rendered markdown, so all the user sees is the output of bmdf seq 3
  • bmdf formats output as a "Bash fence" block
  • bmd (and variants) are useful for displaying commands (and their output) in Markdown (especially in conjunction with mdcmd).
• mdcmd is idempotent:
  • It looks for the block immediately following the <!-- `[cmd...]` --> line, and replaces that with the output of running [cmd...].
  • If there's already output there, it will be replaced with new/current output.

HTML example

Scripts that output raw HTML also work, e.g. print-table.py generates this table:

header 1	header 2
cell 1	cell 2

That table is generated by a line like:

<!-- `python test/print-table.py` -->

mdcmd maintains an output block immediately after it:

<!-- `python test/print-table.py` -->
<table>
  <tr>
    <th>header 1</th>
    <th>header 2</th>
  </tr>
  <tr>
    <td>cell 1</td>
    <td>cell 2</td>
  </tr>
</table>

bmd: format bash command and output as Markdown

bmd --help

Usage: bmd [OPTIONS] COMMAND...

  Format a command and its output to markdown, either in a `bash`-fence or
  <details> block, and copy it to the clipboard.

Options:
  -A, --strip-ansi                Strip ANSI escape sequences from output
  -C, --no-copy                   Disable copying output to clipboard
                                  (normally uses first available executable
                                  from ['pbcopy', 'xclip', 'clip']
  -e, --error-fmt TEXT            If the wrapped command exits non-zero,
                                  append a line of output formatted with this
                                  string. One "%d" placeholder may be used,
                                  for the returncode. Defaults to
                                  $BMDF_ERR_FMT
  -E, --env TEXT                  k=v env vars to set, for the wrapped command
  -f, --fence                     Pass 0-3x to configure output style: 0x:
                                  print output lines, prepended by "# "; 1x:
                                  print a "```bash" fence block including the
                                  <command> and commented output lines; 2x:
                                  print a bash-fenced command followed by
                                  plain-fenced output lines; 3x: print a
                                  <details/> block, with command <summary/>
                                  and collapsed output lines in a plain fence.
                                  Equivalent to -y/--style
                                  {comment,bash,split,details} respectively;
                                  passing both with conflicting styles is an
                                  error.
  -i, --include-stderr / -I, --no-include-stderr
                                  Capture and interleave both stdout and
                                  stderr streams; falls back to
                                  $BMDF_INCLUDE_STDERR
  -s, --shell / -S, --no-shell    Disable "shell" mode for the command; falls
                                  back to $BMDF_SHELL, but defaults to True if
                                  neither is set
  -t, --fence-type TEXT           When -f/--fence is 2 or 3, this customizes
                                  the fence syntax type that the output is
                                  wrapped in
  -r, --exit-code INTEGER         Expected exit code; bmdf exits 0 if command
                                  exits with this code, non-zero otherwise
                                  (useful for diff commands that exit 1 on
                                  differences)
  -u, --expanduser / -U, --no-expanduser
                                  Pass commands through `os.path.expanduser`
                                  before `subprocess`; falls back to
                                  $BMDF_EXPANDUSER
  -v, --expandvars / -V, --no-expandvars
                                  Pass commands through `os.path.expandvars`
                                  before `subprocess`; falls back to
                                  $BMDF_EXPANDVARS
  -w, --workdir TEXT              `cd` to this directory before executing
                                  (falls back to $BMDF_WORKDIR
  -x, --executable TEXT           `shell_executable` to pass to Popen
                                  pipelines (default: $SHELL)
  -y, --style TEXT                Named output style, an alternative to
                                  counting -f/--fence: "comment" (=0x -f),
                                  "bash" (=1x), "split" (=2x), "details"
                                  (=3x), or "console" (a ```console fence with
                                  a "$ "-prefixed command and raw, uncommented
                                  output). A unique prefix or (failing that)
                                  substring also works (e.g. "con"/"sole" ->
                                  console). Passing both -f/--fence and
                                  -y/--style with conflicting styles is an
                                  error.
  --help                          Show this message and exit.

bmd (and aliases bmdf, bmdff, bmdfff, bmdc) takes a bash command as input, and renders the command and/or its output in various Markdown-friendly formats.

The output style can be selected two ways:

• counting -f/--fence (0–3×), or
• naming it with -y/--style ({comment, bash, split, details, console})

The aliases map to specific styles:

Alias	Equivalent	Style
bmd	bmd -y comment (-f ✕0)	output lines, # -commented
bmdf	bmd -y bash (-f)	one bash fence: command + commented output
bmdff	bmd -y split (-ff)	bash fence for command, plain fence for output
bmdfff	bmd -y details (-fff)	<details> block with command <summary>
bmdc	bmd -y console	console fence: $ -prefixed command + raw output

(console has no -f count; it's reachable only via -y console or bmdc.)

bmdf (bmd -f): command+output mode

Suppose you want to embed a command and its output in a README.md, like this:

seq 3
# 1
# 2
# 3

(Note how the command is bash-highlighted, and output lines are rendered as comments)

Put a placeholder like this in your README.md:

<!-- `bmdf seq 3` -->

then run mdcmd to update your README containing this embedded command block.

bmdff (bmd -ff): two-fence mode

bmdff (alias for bmd -ff) renders two code fences, one with the Bash command (syntax-highlighted appropriately), and a second (non-highlighted) block with the output, e.g.:

<!-- `bmdff seq 5` -->

becomes:

seq 5

1
2
3
4
5

bmdfff (bmd -fff): <details> mode

When a command's output is large, rendering it as a <details><summary> (with the output collapsed, by default) may be preferable.

bmdfff (3 fs, alias for bmd -fff) transforms placeholders like this:

<!-- `bmdfff seq 10` -->

to:

seq 10

1
2
3
4
5
6
7
8
9
10

bmdc (bmd -y console): <console> mode

bmdc (alias for bmd -y console) renders a single ```console fence, with the command on a $ -prefixed prompt line and its output left raw (uncommented), e.g.:

<!-- `bmdc echo yay` -->

becomes:

$ echo yay
yay

(-t/--fence-type overrides the console fence language, e.g. for shell-session.)

Piping

Piping works too, e.g.:

<!-- `bmdf -- seq 10 | wc -l` -->

will become:

seq 10 | wc -l
# 10

(the -- is needed so that that -l isn't parsed as an opt to bmdf)

Env vars

By default, shell=True is passed to subprocess calls (but can be disabled via -S).

This means env vars are expanded; they can also be set via -E, e.g.:

<!-- `bmdf -E FOO=bar echo $FOO` -->

yields:

FOO=bar echo '$FOO'
# bar

More examples of quoting/splitting behavior

Quoting "$FOO":

<!-- `bmdf -E FOO=bar echo "$FOO"` -->

yields:

FOO=bar echo '$FOO'
# bar

Arg with spaces:

<!-- `bmdf -E FOO=bar echo "FOO: $FOO"` -->

yields:

FOO=bar echo 'FOO: $FOO'
# FOO: bar

Escaping $:

<!-- `bmdf -E FOO=bar echo "\$FOO=$FOO"` -->

yields:

FOO=bar echo '\$FOO=$FOO'
# $FOO=bar

-w/--workdir / $BMDF_WORKDIR

By default, bmdf runs in the current working directory. This can be overridden with -w:

<!-- `bmdf -w .github ls` -->

ls
# workflows

toc: Markdown Table of Contents

toc --help

Usage: toc [OPTIONS] [PATH]

  Generate a table of contents from a markdown file.

  If no PATH is provided, will try to use $MDCMD_FILE (set by mdcmd), or
  default to README.md if that's not set.

Options:
  -n, --indent-size INTEGER  Indent size (spaces)
  --help                     Show this message and exit.

toc generates a table of contents from markdown headings, and pairs well with mdcmd for maintaining TOCs in markdown files.

1. Put a line like this in your README.md:

   <!-- `toc` -->
   
   

   (the trailing blank line is important, don't put other content immediately under a <!-- ... --> line)

2. Put empty <a> tags next to headings to includ them in the TOC (and specify an id):

   ## My section heading <a id="my-section"></a>
   

   This allows for custom/short ids, as well as skipping sections.

3. Run mdcmd as usual:

   # Update all command blocks in README.md, including the TOC
   mdcmd
   

   mdcmd will see the <!-- toc -->, and embed the TOC generated by toc under it.

A mktoc script is also provided, which just wraps mdcmd -x '^toc$' (mktoc was implemented separately, in previous versions, before being decomposed into mdcmd and toc in 0.7.0).

Examples

• The examples in this file are all rendered by bmdf and mdcmd.
• The TOC above is rendered by toc.
• The ci.yml GitHub Action verifies the examples and TOC.

These repos' READMEs also use bmdf / mdcmd / toc to execute example commands (and in some cases also verify them with a GitHub Action):

• runsascoded/juq
• runsascoded/utz
• runsascoded/dvc-utils
• ryan-williams/dvc-helpers (GHA)
• ryan-williams/git-helpers
• ryan-williams/parquet-helpers (GHA)
• ryan-williams/tdbs-dask
• TileDB-Inc/scverse-ml-workshop-2024

GitHub Action

A reusable GitHub Action is available on the v1 branch:

- uses: runsascoded/mdcmd@v1

It runs mdcmd (and optionally mktoc) and fails if the files are out of date.

Inputs

Input	Description	Default
version	mdcmd version to install	latest
files	Space-separated files to process	README.md
run-mktoc	Run mktoc for TOC generation	auto-detect*
fail-on-diff	Fail if changes detected	true

* Auto-detects if files contain <!-- \toc` -->` marker

Example

See use-prms workflow (runs) for a real-world example.