databricks-notebook-linter 0.1.0

Pre-commit hook to fix bare magic commands in Databricks .py-format notebooks

databricks-notebook-linter

A pre-commit hook that fixes bare magic commands in Databricks .py-format notebooks.

Problem

Databricks exports notebooks as .py files with special comment markers. Magic commands like %pip install and !nvidia-smi appear as bare lines, which are invalid Python syntax. This breaks linters (ruff, flake8) and type checkers (ty, mypy) that try to parse these files.

Solution

This tool prefixes bare magic commands with # MAGIC, converting them to Python comments that Databricks still recognizes and executes:

# Before
%pip install some-package==1.0.4

# After
# MAGIC %pip install some-package==1.0.4

It handles:

• Single-line magic commands (%pip, %sql, %md, %sh, %fs, %run, %python, %r, %scala)
• Shell bang commands (!nvidia-smi)
• Multiline continuations (%pip install -U \)
• Block-level magic -- if a %pip or ! command is inside an if, for, try, or other block, the entire block is prefixed
• Nested blocks -- magic three levels deep prefixes all enclosing levels
• Compound blocks -- if/elif/else, try/except/finally treated as single units
• Mixed cells -- regular Python lines outside blocks are left untouched

The tool is idempotent -- running it twice produces the same result.

Usage

As a pre-commit hook

Add to your .pre-commit-config.yaml:

repos:
  - repo: https://github.com/yipitdata/databricks-notebook-linter
    rev: v0.1.0
    hooks:
      - id: fix-databricks-magic

By default the hook runs in check mode -- it reports unfixed magic commands and fails without modifying files. To auto-fix files on commit, pass --fix:

hooks:
  - id: fix-databricks-magic
    args: [--fix]

As a CLI tool

pip install databricks-notebook-linter

# Check mode (default): report issues, exit 1 if any found
fix-databricks-magic path/to/notebook.py

# Fix mode: rewrite files in place, exit 1 if any changed
fix-databricks-magic --fix path/to/notebook.py

Check mode output

notebook.py:5: bare magic command '%pip install foo' needs '# MAGIC' prefix
notebook.py:10: line in block containing magic needs '# MAGIC' prefix

How it works

1. Checks if the file starts with # Databricks notebook source -- skips non-notebook files
2. Splits the file into cells on # COMMAND ---------- boundaries
3. For each cell, scans for bare magic lines (lines starting with %pip, !, etc.)
4. If magic is at the top level, marks just that line (and any continuation lines)
5. If magic is indented inside a block, walks backwards to find the top-level enclosing block and forwards to find the end of compound blocks (else, except, finally), then marks every line in the block
6. Prefixes all marked lines with # MAGIC, preserving relative indentation for block-internal lines

Development

make setup    # install dependencies
make test     # run tests (with 100% branch coverage enforcement)
make lint     # run ruff
make format   # auto-format

Releasing

# All at once: tag, publish to PyPI, push
make release VERSION=x.y.z

# Or in two steps:
make tag-release VERSION=x.y.z    # bump, commit, tag (local only)
make push-release VERSION=x.y.z   # build, publish to PyPI, push commit + tag

tag-release validates a clean working tree on main, runs tests and lint, bumps the version in pyproject.toml and README.md, commits, and creates an annotated tag. push-release builds, publishes to PyPI, then pushes. Nothing reaches the remote until the PyPI publish succeeds.

License

MIT