synapse-controller 0.1.0

CLI and backend for driving Azure Synapse Analytics pipelines and notebooks from a single config.yaml.

synapse-controller

A Python CLI + backend that drives Azure Synapse Analytics from a single config.yaml. Wraps azure-synapse-artifacts to start, monitor, list, and cancel pipeline runs, and to run / debug notebooks against a Spark session.

See ARCHITECTURE.md for the architectural source of truth: stack, layout, config shape, CLI behavior, and project conventions.

Quickstart

uv sync --extra dev
uv run python -m synapse_controller --help

Copy config.example.yaml to config.yaml, fill in your workspace identity, then:

uv run python -m synapse_controller --config config.yaml --profile dev pipelines list

Note — invoke as python -m synapse_controller, not as a script entry point. Defender on Belastingdienst-managed Windows blocks executing binaries from .venv\Scripts\, so all tools (black, isort, flake8, mypy, pytest, this CLI) are launched via python -m.

Notebook output streaming caveat — notebooks run and notebooks debug execute a workspace notebook by submitting each code cell to a Livy Spark session. Synapse's Livy returns a cell's captured output only when the cell finishes, so per-line stdout during a long-running cell is not available. Feedback granularity is one cell.

Development

uv run python -m black .
uv run python -m isort .
uv run python -m flake8 src tests
uv run python -m mypy src tests
uv run python -m pytest

Authentication is always DefaultAzureCredential. Make sure you are signed in (e.g. az login) before running commands that touch the workspace.