bmdf 0.6.2

Execute commands in Markdown files, embed output, generate TOCs

bmdf

Bash markdown fences: execute commands in Markdown files, embed output, generate TOCs

e.g.:

seq 3
# 1
# 2
# 3

☝️ This block is updated programmatically (by mdcmd), and verified in CI.

• Overview
• Install
• mdcmd: execute commands in Markdown files, embed output
  • bmdf example
  • HTML example
• bmd: format bash command and output as Markdown
  • bmdf (bmd -f): command+output mode
  • bmdff (bmd -ff): two-fence mode
  • bmdfff (bmd -fff): <details> mode
  • Piping
  • Env vars
  • -w/--workdir / $BMDF_WORKDIR
• mktoc: Markdown Table of Contents
• Examples

☝️ This TOC is generated programmatically (by mktoc), and verified in CI.

Overview

This package provides 3 CLIs:

• mdcmd: execute shell commands in Markdown files, embed output
• bmd: run Bash commands, wrap output for Markdown embedding
  • Useful in conjunction with mdcmd
  • bmdf, bmdff, bmdfff provide different types of "fencing" for command output
• mktoc: update Markdown table of contents (with custom "id"s for sections)

Install

Global install via pipx or uv (recommended):

pipx install bmdf
# or: uv tool install bmdf

You can also install in the current (v)env:

pip install bmdf

mdcmd: execute commands in Markdown files, embed output

mdcmd --help

Usage: mdcmd [OPTIONS] [PATH] [OUT_PATH]

  Parse a Markdown file, updating blocks preceded by <!-- `[cmd...]` -->
  delimiters.

  If no paths are provided, will look for a README.md, and operate "in-place"
  (same as ``mdcmd -i README.md``).

Options:
  -a, --amend                     Squash changes onto the previous Git commit;
                                  suitable for use with `git rebase -x`
  -C, --no-concurrent             Run commands in sequence (by default, they
                                  are run concurrently)
  -i, --inplace / -I, --no-inplace
                                  Edit the file in-place
  -n, --dry-run                   Print the commands that would be run, but
                                  don't execute them
  -T, --no-cwd-tmpdir             In in-place mode, use a system temporary-
                                  directory (instead of the current workdir,
                                  which is the default)
  -x, --execute TEXT              Only execute commands that match these
                                  regular expressions
  -X, --exclude TEXT              Only execute commands that don't match these
                                  regular expressions
  --help                          Show this message and exit.

# Modify README.md in-place
mdcmd -i README.md
# Same as above; no args defaults to `-i README.md`
mdcmd

That's how the various command examples in this file are generated / updated!

bmdf example

The example at the top of this file is generated by a line like:

<!-- `bmdf seq 3` -->

mdcmd transforms that into:

<!-- `bmdf seq 3` -->
```bash
seq 3
# 1
# 2
# 3
```

Notes:

• HTML comments (<!-- ... -->) are hidden in rendered markdown, so all the user sees is the output of bmdf seq 3
  • bmdf formats output as a "Bash fence" block
  • bmd (and variants) are useful for displaying commands (and their output) in Markdown (especially in conjunction with mdcmd).
• mdcmd is idempotent:
  • It looks for the block immediately following the <!-- `[cmd...]` --> line, and replaces that with the output of running [cmd...].
  • If there's already output there, it will be replaced with new/current output.

HTML example

Scripts that output raw HTML also work, e.g. print-table.py generates this table:

header 1	header 2
cell 1	cell 2

That table is generated by a line like:

<!-- `python test/print-table.py` -->

mdcmd maintains an output block immediately after it:

<!-- `python test/print-table.py` -->
<table>
  <tr>
    <th>header 1</th>
    <th>header 2</th>
  </tr>
  <tr>
    <td>cell 1</td>
    <td>cell 2</td>
  </tr>
</table>

bmd: format bash command and output as Markdown

bmd --help

Usage: bmd [OPTIONS] COMMAND...

  Format a command and its output to markdown, either in a `bash`-fence or
  <details> block, and copy it to the clipboard.

Options:
  -A, --strip-ansi                Strip ANSI escape sequences from output
  -C, --no-copy                   Disable copying output to clipboard
                                  (normally uses first available executable
                                  from ['pbcopy', 'xclip', 'clip']
  -e, --error-fmt TEXT            If the wrapped command exits non-zero,
                                  append a line of output formatted with this
                                  string. One "%d" placeholder may be used,
                                  for the returncode. Defaults to
                                  $BMDF_ERR_FMT
  -E, --env TEXT                  k=v env vars to set, for the wrapped command
  -f, --fence                     Pass 0-3x to configure output style: 0x:
                                  print output lines, prepended by "# "; 1x:
                                  print a "```bash" fence block including the
                                  <command> and commented output lines; 2x:
                                  print a bash-fenced command followed by
                                  plain-fenced output lines; 3x: print a
                                  <details/> block, with command <summary/>
                                  and collapsed output lines in a plain fence.
  -i, --include-stderr / -I, --no-include-stderr
                                  Capture and interleave both stdout and
                                  stderr streams; falls back to
                                  $BMDF_INCLUDE_STDERR
  -s, --shell / -S, --no-shell    Disable "shell" mode for the command; falls
                                  back to $BMDF_SHELL, but defaults to True if
                                  neither is set
  -t, --fence-type TEXT           When -f/--fence is 2 or 3, this customizes
                                  the fence syntax type that the output is
                                  wrapped in
  -u, --expanduser / -U, --no-expanduser
                                  Pass commands through `os.path.expanduser`
                                  before `subprocess`; falls back to
                                  $BMDF_EXPANDUSER
  -v, --expandvars / -V, --no-expandvars
                                  Pass commands through `os.path.expandvars`
                                  before `subprocess`; falls back to
                                  $BMDF_EXPANDVARS
  -w, --workdir TEXT              `cd` to this directory before executing
                                  (falls back to $BMDF_WORKDIR
  -x, --executable TEXT           `shell_executable` to pass to Popen
                                  pipelines (default: $SHELL)
  --help                          Show this message and exit.

bmd (and aliases bmdf, bmdff, bmdfff) takes a bash command as input, and renders the command and/or its output in various Markdown-friendly formats:

bmdf (bmd -f): command+output mode

Suppose you want to embed a command and its output in a README.md, like this:

seq 3
# 1
# 2
# 3

(Note how the command is bash-highlighted, and output lines are rendered as comments)

Put a placeholder like this in your README.md:

<!-- `bmdf seq 3` -->

then run mdcmd to update your README containing this embedded command block.

bmdff (bmd -ff): two-fence mode

bmdff (alias for bmd -ff) renders two code fences, one with the Bash command (syntax-highlighted appropriately), and a second (non-highlighted) block with the output, e.g.:

<!-- `bmdff seq 5` -->

becomes:

seq 5

1
2
3
4
5

bmdfff (bmd -fff): <details> mode

When a command's output is large, rendering it as a <details><summary> (with the output collapsed, by default) may be preferable.

bmdfff (3 fs, alias for bmd -fff) transforms placeholders like this:

<!-- `bmdfff seq 10` -->

to:

seq 10

1
2
3
4
5
6
7
8
9
10

Piping

Piping works too, e.g.:

<!-- `bmdf -- seq 10 | wc -l` -->

will become:

seq 10 | wc -l
# 10

(the -- is needed so that that -l isn't parsed as an opt to bmdf)

Env vars

By default, shell=True is passed to subprocess calls (but can be disabled via -S).

This means env vars are expanded; they can also be set via -E, e.g.:

<!-- `bmdf -E FOO=bar echo $FOO` -->

yields:

FOO=bar echo '$FOO'
# bar

More examples of quoting/splitting behavior

Quoting "$FOO":

<!-- `bmdf -E FOO=bar echo "$FOO"` -->

yields:

FOO=bar echo '$FOO'
# bar

Arg with spaces:

<!-- `bmdf -E FOO=bar echo "FOO: $FOO"` -->

yields:

FOO=bar echo 'FOO: $FOO'
# FOO: bar

Escaping $:

<!-- `bmdf -E FOO=bar echo "\$FOO=$FOO"` -->

yields:

FOO=bar echo '\$FOO=$FOO'
# $FOO=bar

-w/--workdir / $BMDF_WORKDIR

By default, bmdf runs in the current working directory. This can be overridden with -w:

<!-- `bmdf -w .github ls` -->

ls
# workflows

mktoc: Markdown Table of Contents

mktoc --help

Usage: mktoc [OPTIONS] [PATH] [OUT_PATH]

  Insert a table of contents (TOC) in a markdown file.

  Looks for a pair of sentinel lines to insert or update the TOC between:
  ``<!-- toc -->``, ``<!-- /toc -->``.

  If an empty line follows the opening ``<!-- toc -->`` line, the TOC will be
  inserted there (along with the closing sentinel); this is useful when
  initially generating a TOC.

  If no ``out_path`` is provided, will operate "in-place" on ``README.md`` (as
  if ``mktoc -i README.md`` was passed).

Options:
  -a, --amend                     Squash changes onto the previous Git commit;
                                  suitable for use with `git rebase -x`
  -i, --inplace / -I, --no-inplace
                                  Edit the file in-place
  -n, --indent-size INTEGER       Indent size (spaces)
  -T, --no-cwd-tmpdir             In in-place mode, use a system temporary-
                                  directory (instead of the current workdir,
                                  which is the default)
  --help                          Show this message and exit.

Put a block like this in your README.md:

<!-- toc -->
<!-- /toc -->

Then put empty <a> tags next to the headings you want to include, e.g.:

## My section heading <a id="my-section"></a>

(This allows for customizing and shortening the ids, as well as skipping sections)

Then run:

# Modify README.md in-place
mktoc -i README.md
# Same as above; no args defaults to `-i README.md`
mktoc

And the <!-- toc --> section will have a table of contents injected (like the one at the top of this file).

Examples

The examples in this file are rendered using bmdf and mdcmd, and the TOC is rendered using mktoc. Both are verified by the ci.yml GitHub Action.

These repos' READMEs also use bmdf / mdcmd / mktoc to execute/verify examples commands (and in some cases are also verified by GitHub Actions):

• runsascoded/juq
• runsascoded/utz
• runsascoded/dvc-utils
• ryan-williams/dvc-helpers (GHA)
• ryan-williams/git-helpers
• ryan-williams/parquet-helpers (GHA)
• ryan-williams/tdbs-dask
• TileDB-Inc/scverse-ml-workshop-2024