pytest-bashdoctest

A pytest plugin for testing bash command examples in markdown documentation.

Why pytest-bashdoctest?

API documentation gets outdated fast. Manually testing curl examples is tedious. This plugin automatically tests bash code blocks in your markdown files against actual API responses, keeping docs and code in sync.

Installation

pip install pytest-bashdoctest

Or with uv:

uv add pytest-bashdoctest

Quick Start

1. Write testable examples

In your API.md:

## User API

```bash
$ curl -s "https://api.github.com/users/dnouri"
{
  "login": "dnouri",
  ...
  "name": "Daniel Nouri",
  ...
  "bio": "Machine Learning and Programming",
  ...
}
```

The ... pattern matches any content, perfect for skipping dynamic or irrelevant fields.

2. Run tests

pytest --bashdoctest API.md -v

Your documentation is now executable and verified.

Note: The --bashdoctest flag is required to enable bash doctest collection. This prevents conflicts with other pytest plugins that also process markdown files (like pytest-markdown-docs).

ELLIPSIS Patterns

Line-Level: Skip Blocks

Use standalone ... to skip entire sections:

```bash
$ curl -s "https://api.github.com/users/dnouri"
{
  "login": "dnouri",
  ...
  "type": "User",
  ...
  "bio": "Machine Learning and Programming",
  ...
}
```

Show the fields that matter, skip the rest.

String-Level: Partial Matching

Use ... inside strings for dynamic values:

```bash
$ curl -s "https://api.github.com/users/dnouri"
{
  "login": "dnouri",
  ...
  "avatar_url": "https://avatars.githubusercontent.com/u/...?v=4",
  ...
  "created_at": "2009-...-...T...Z",
  ...
}
```

Useful for URLs, UUIDs, timestamps, and other values that change but follow a pattern.

Collection-Level: Objects and Arrays

Use {...} and [...] to match entire structures:

```bash
$ curl -s "https://api.github.com/repos/dnouri/nolearn"
{
  ...
  "name": "nolearn",
  ...
  "owner": {...},
  ...
  "license": {...},
  ...
}
```

Focus on the structure you care about without showing every detail.

Combining Patterns

Mix all three for flexible matching:

```bash
$ curl -s "https://api.github.com/users/dnouri"
{
  "login": "dnouri",
  ...
  "avatar_url": "https://avatars.githubusercontent.com/...",
  ...
  "bio": "Machine Learning and Programming",
  ...
  "public_repos": ...,
  ...
}
```

Configuration

Specifying Files to Test

Explicit paths:

pytest --bashdoctest README.md -v
pytest --bashdoctest docs/api.md -v

Via configuration in pyproject.toml:

[tool.pytest.ini_options]
testpaths = ["tests", "README.md", "docs"]

Then pytest --bashdoctest will test both your test suite and documentation.

Environment Variables

The Problem: Your bash examples often need API keys, custom URLs, or other configuration. Hard-coding these in documentation is insecure and inflexible.

The Solution: The bashdoctest_env fixture. This session-scoped fixture provides environment variables to all bash examples in your markdown files.

When you DON'T need it: If your bash examples only use public APIs without authentication (like the GitHub examples in this README), you don't need to define this fixture. The plugin provides a default empty fixture.

When you DO need it: If your examples need API keys, custom URLs, or other config, create conftest.py:

import os
import pytest

@pytest.fixture(scope="session")
def bashdoctest_env():
    """Environment variables for bash documentation examples."""
    api_key = os.getenv("API_KEY")
    if not api_key:
        pytest.skip("API_KEY not set - skipping bash doctest examples")

    return {
        "API_KEY": api_key,
        "API_URL": os.getenv("API_URL", "https://api.example.com"),
    }

Use in your markdown files:

```
$ curl -s "$API_URL/users" -H "Authorization: Bearer $API_KEY"
{
  "users": [...]
}
```

The environment variables from your bashdoctest_env fixture are merged with os.environ when executing commands, so $API_KEY and $API_URL will be available to all bash examples.

How It Works

1. Plugin activates when you pass --bashdoctest flag
2. Collects markdown files you specify (explicit paths or via testpaths)
3. Parser extracts bash code blocks (only files with actual bash examples)
4. Each bash block becomes a test item
5. Commands execute with your bashdoctest_env variables merged into environment
6. Output matches against expected using ELLIPSIS rules

Architecture:

src/pytest_bashdoctest/
   parser.py      # Extract bash blocks
   matcher.py     # ELLIPSIS matching
   executor.py    # Run commands
   formatter.py   # Format failures
   plugin.py      # Pytest hooks

Core modules are pure Python (no pytest dependency) for portability.

Testing the Plugin

# Run unit tests
pytest tests/ -v

# Test with coverage
pytest tests/ --cov=pytest_bashdoctest --cov-report=term-missing

# Test this README (dogfooding!)
pytest --bashdoctest README.md -v

Limitations

• Commands timeout after 30 seconds (configurable in executor.py)
• Interactive commands will hang
• Uses shell=True (commands come from your trusted markdown files)
• Output buffering may differ for very large responses

Development

Setup

git clone https://github.com/dnouri/pytest-bashdoctest.git
cd pytest-bashdoctest

# Install dependencies
uv sync --dev

# Install pre-commit tool globally (recommended, one-time setup)
uv tool install pre-commit

# Install git hooks (runs automatically on every commit)
pre-commit install

Running Tests

# Run unit tests
uv run pytest tests/ -v

# Test README examples
uv run pytest --bashdoctest README.md -v

# Run with coverage
uv run pytest tests/ --cov=pytest_bashdoctest --cov-report=term-missing

Code Quality

Pre-commit hooks automatically run on every commit to ensure code quality. To run manually:

# Run all pre-commit hooks (after uv tool install pre-commit)
pre-commit run --all-files

# Or using project dependencies
uv run pre-commit run --all-files

# Run specific checks
uv run ruff check .              # Linting
uv run ruff format .             # Auto-format code
uv run ruff check --fix .        # Auto-fix linting issues

To skip pre-commit hooks (not recommended):

git commit --no-verify -m "message"

Releasing

Update version in pyproject.toml, then:

git tag v2025.10.4 && git push --tags

Publishes to PyPI automatically via GitHub Actions.

License

MIT License - see LICENSE file for details.