docsub 0.8.1

Embed text and data into Markdown files

docsub

Substitute dynamically generated content in Markdown files

[!WARNING]

• With docsub, every documentation file may become executable.
• Never use docsub to process files from untrusted sources.
• This project is in experimental state, syntax and functionality may change significantly.
• If still want to try it, use pinned package version docsub==0.8.1

Features

• Embed static files
• Embed command execution results
• Idempotent substitutions
• Invisible non-intrusive markup using comment blocks
• Plays nicely with other markups
• Extensible with project-local commands
• Configurable with config files and env vars

Use cases

• Manage partially duplicate docs for multiple destinations
• Manage docs for monorepositories
• Embed CLI reference in docs
• Embed dynamically generated content:
  • Project metadata
  • Test reports
  • Models evaluation results

[!NOTE] This file uses docsub itself. Dig into raw markup if interested.

Docsub is not...

• ...a documentation engine like Sphinx or MkDocs
• ...a full-featured static website generator like Pelican
• ...a templating engine like Jinja
• ...a replacement for Bump My Version

Usage

$ uv run docsub apply -i README.md

From separate files...

README.md # Title <!-- docsub: begin --> <!-- docsub: include info.md --> <!-- docsub: include features.md --> ... <!-- docsub: end --> ## Table <!-- docsub: begin --> <!-- docsub: include data.md --> <!-- docsub: lines after 2 --> | Col 1 | Col 2 | |-------|-------| ... <!-- docsub: end --> ## Code <!-- docsub: begin #code --> <!-- docsub: include func.py --> <!-- docsub: lines after 1 upto -1 --> ```python ... ``` <!-- docsub: end #code -->

info.md > Long description. features.md * Feature 1 * Feature 2 * Feature 3 data.md | Key 1 | value 1 | | Key 2 | value 2 | | Key 3 | value 3 | func.py def func(): pass

Get merged document

and keep it updated!

# Title
<!-- docsub: begin -->
<!-- docsub: include info.md -->
<!-- docsub: include features.md -->
> Long description.
* Feature 1
* Feature 2
* Feature 3
<!-- docsub: end -->

## Table
<!-- docsub: begin -->
<!-- docsub: include data.md -->
<!-- docsub: lines after 2 -->
| Col 1 | Col 2 |
|-------|-------|
| Key 1 | value 1 |
| Key 2 | value 2 |
| Key 3 | value 3 |
<!-- docsub: end -->

## Code
<!-- docsub: begin #code -->
<!-- docsub: include func.py -->
<!-- docsub: lines after 1 upto -1 -->
```python
def func():
    pass
```
<!-- docsub: end #code -->

Installation

Development dependency

Recommended. The most flexible installation option, allowing project-local commands to utilize project codebase.

# pyproject.toml
[dependency-groups]
dev = [
  "docsub==0.8.1",
]

Global installation

Works for simple cases.

uv tool install docsub==0.8.1

Syntax

The syntax is purposefully verbose. This is fine, you are not supposed to edit it often. But it's searchable and sticks in eye when scrolling down large documents.

Docsub uses line-based substitution syntax based on directives and substitution blocks.

Markdown

Directive

Markdown directive is one-line comment:

<!-- docsub: <directive> [directive args] -->

There are multiple directive types.

Substitution block

Markdown substitution block is a sequence of lines, starting with begin directive and ending with end directive.

<!-- docsub: begin -->
<!-- docsub: help docsub -->
<!-- docsub: include CHANGELOG.md -->
Inner text will be replaced.
<!-- docsub: this whole line is treated as plain text -->
This text will be replaced too.
<!-- docsub: end -->

One or many other directives must come at the top of the block, otherwise they are treated as plain text. Blocks without producing directives are not allowed. Block's inner text will be replaced upon substitution, unless modifier directives are used, e.g. lines.

If docsub substitution block lies inside markdown fenced code block, it is not substituted (example: fenced code blocks above and below this paragraph, see the raw markup). To put dynamic content into a fenced code block, place begin and end around it and use lines after N upto -M (example: Usage section).

For nested blocks, only top level substitution is performed. Use block #identifier to distinguish between nesting levels.

<!-- docsub: begin #top -->
<!-- docsub: include part.md -->
<!-- docsub: begin -->
<!-- docsub: include nested.md -->
<!-- docsub: end -->
<!-- docsub: end #top -->

Directives

• Block delimiters: begin, end
• Producing commands: exec, help, include, x
• Modifying commands: lines, strip

begin

begin [#identifier]

Open substitution target block. To distinguish between nesting levels, use block #identifier, starting with #.

end

end [#identifier]

Close substitution target block.

exec

exec <shell commands>

Execute <shell commands> with sh -c and substitute stdout. Allows pipes and other shell functionality. If possible, avoid using this directive.

• cmd.exec.work_dir — shell working directory, default '.'
• cmd.exec.env_vars — dict of additional environment variables, default {}

help

help <command> [subcommand...]
help python -m <command> [subcommand...]

Display help for CLI utility or Python module. Use this command to document CLI instead of exec. Runs command [subcommand...] --help or python -m command [subcommand...] --help respectively. Directive args must be a space-separated sequence of characters [-._a-zA-Z0-9].

• cmd.help.env_vars — dict of additional environment variables, default {}

include

include path/to/file

Literally include file specified by path relative to base_dir config option.

• cmd.include.base_dir — base directory for relative paths

lines

lines [after N] [upto -M]

Upon substitution, keep original target block lines: first N and/or last M. Only one lines command is allowed inside the block.

strip

strip

Strip whitespace in substitution result:

• initial and trailing blank lines
• trailing whitespace on every line

x

x <project-command> [args and --options]

Execute project-local command declared in docsubfile.py in project root. The naming is inspired by X- HTTP headers and x- convention for reusable YAML sections.

• cmd.x.docsubfile — path to file with project-local commands, absolute or relative to project root (default: docsubfile.py)

Project-local commands

When project root contains file docsubfile.py with commands defined as in example below, they can be used in docsub: x directive. Project commands must be defined as click command and gathered under x group. There is no need to install click separately as docsub depends on it.

If docsub is installed globally and called as uvx docsub, project commands in docsubfile.py have access to docsub dependencies only: click, loguru, rich (see docsub's pyproject.toml for details).

If docsub is installed as project dev dependency and called as uv run docsub, user commands also have access to project modules and dev dependencies. This allows more flexible scenarios.

Project command author can get access to docsub Environment object (including command configs) from click context object (see example below). The docsub Environment object has some useful methods (not documented yet).

Example

$ uv run docsub apply -i sample.md

sample.md

<!-- docsub: begin -->
<!-- docsub: x say-hello Alice Bob -->
Hi there, Alice!
Hi there, Bob!
<!-- docsub: end -->

docsubfile.py

from docsub import Environment, click, pass_env

@click.group()
def x():
    pass

@x.command()
@click.argument('users', nargs=-1)
def say_hello(users: tuple[str, ...]) -> None:
    for user in users:
        click.echo(f'Hi there, {user}!')

@x.command()
@click.argument('users', nargs=-1)
@pass_env
def log_hello(env: Environment, users: tuple[str, ...]) -> None:
    base = env.get_temp_dir('log_hello')
    (base / 'hello.log').write_text(f'said hello to {users}')

Calling project-local commands

Docsub exposes x as CLI command, letting project commands to be executed with project settings:

$ uv run docsub x say-hello Alice Bob
Hi there, Alice!
Hi there, Bob!

Configuration

Configuration resolution order

• command line options (to be documented)
• environment variables (to be documented)
• docsub.toml config file in current working directory
• pyproject.toml, section [tool.docsub] (to be implemented)
• default config values

Root settings

• local_dir — internal working directory at the project root (default: .docsub)

Command settings

See Commands.

Environment variables

(to be documented)

Command line options

(to be documented)

Complete config example

All config keys are optional.

local_dir = ".docsub"  # default

[logging]
#level = "DEBUG"  # default: missing, logging disabled

[cmd.exec]
env_vars = {}  # default
work_dir = "."  # default

[cmd.help.env_vars]
COLUMNS = "60"  # more compact

[cmd.include]
base_dir = "."  # default

[cmd.x]
docsubfile = "docsubfile.py"  # default

[!WARNING] In future releases config keys will be moved under [tool.docsub] root for both pyproject.toml and docsub.toml, this will be a breaking change.

Logging

Docsub uses loguru for logging. Logging is disabled by default. To enable logging, set config option level to one of logging levels supported by loguru.

(logging is rudimentary at the moment)

CLI Reference

$ docsub --help
Usage: python -m docsub [OPTIONS] COMMAND [ARGS]...

╭─ Options ──────────────────────────────────────────────────────────╮
│ --config-file           -c  PATH                                   │
│ --local-dir             -l  PATH                                   │
│ --cmd-exec-work-dir         PATH                                   │
│ --cmd-exec-env-vars         TEXT                                   │
│ --cmd-help-env-vars         TEXT                                   │
│ --cmd-include-base-dir      PATH                                   │
│ --cmd-x-docsubfile      -x  PATH                                   │
│ --version                         Show the version and exit.       │
│ --help                            Show this message and exit.      │
╰────────────────────────────────────────────────────────────────────╯
╭─ Commands ─────────────────────────────────────────────────────────╮
│ apply    Update Markdown files with embedded content.              │
│ x        Project-local commands.                                   │
╰────────────────────────────────────────────────────────────────────╯

docsub apply

$ docsub apply --help
Usage: python -m docsub apply [OPTIONS] FILES...

Update Markdown files with embedded content.
Read FILES and perform substitutions one by one. If one file depends
on another, place it after that file.

╭─ Options ──────────────────────────────────────────────────────────╮
│ --in-place  -i    Process files in-place                           │
│ --help            Show this message and exit.                      │
╰────────────────────────────────────────────────────────────────────╯

docsub x

$ docsub x --help
Usage: python -m docsub x [OPTIONS] COMMAND [ARGS]...

Project-local commands.

╭─ Options ──────────────────────────────────────────────────────────╮
│ --help      Show this message and exit.                            │
╰────────────────────────────────────────────────────────────────────╯

History

This project appeared to maintain docs for multipython project. You may check it up for usage examples.

Authors

• Michael Makukha

License

MIT License

Changelog

CHANGELOG.md