Cursus: Automatic SageMaker Pipeline Generation

PyPI version Python 3.9+ License: MIT Documentation Status

Turn a pipeline graph plus a JSON config into a complete, production-ready SageMaker pipeline — automatically.

Cursus is a specification-driven pipeline generation system for Amazon SageMaker. You describe your ML workflow as a DAG of step names; Cursus resolves the dependencies between steps, wires their inputs and outputs, looks up each step's declarative interface, and assembles a runnable sagemaker.workflow.pipeline.Pipeline. You say what the pipeline is — Cursus figures out how to build it.

📖 Full documentation: cursus.readthedocs.io (mirror: tianpeiluke.github.io/cursus) — getting-started, tutorials, concepts & architecture, and the complete API / CLI / MCP / step-catalog reference.

Installation

pip install cursus

Requires Python 3.9+ and targets the SageMaker Python SDK 2.x (sagemaker>=2.248.0,<3).

Optional extras keep heavy ML/data libraries out of the core install — pull in only what your steps run:

pip install "cursus[processing]"   # pandas / numpy data-processing utilities
pip install "cursus[pytorch]"      # PyTorch / Lightning
pip install "cursus[gbm]"          # XGBoost / LightGBM
pip install "cursus[nlp]"          # tokenizers / transformers
pip install "cursus[mcp]"          # MCP server for LLM agents (mcp + anyio)
pip install "cursus[all]"          # everything

Verify:

cursus --version
python -c "import cursus; print(cursus.__version__)"

Quick Start

There are three ways in, highest-level first. They all compile through the same engine, so the same DAG + config produces the same pipeline.

Every path needs two ingredients: a DAG (step names + edges) and a config JSON whose metadata.config_types maps each node to a configuration class. Compilation is offline; you only need AWS credentials to deploy (--upsert) or run (--start).

1. Start from the pre-built pipeline catalog

Cursus ships 44 validated DAGs across 8 frameworks. Let the router recommend one and build it:

from cursus.pipeline_catalog import recommend_dag, load_shared_dag
from cursus import PipelineDAGCompiler

# recommend_dag returns a ranked list of matches (dicts with 'id', 'score', ...)
recommendations = recommend_dag(framework="xgboost", task_type="end_to_end")
dag = load_shared_dag(recommendations[0]["id"])

pipeline, report = PipelineDAGCompiler(config_path="config.json").compile_with_report(dag)
print(pipeline.name, "-", len(pipeline.steps), "steps")

2. Compile from the command line

Reproducible, no-glue path — point it at a DAG JSON and a config JSON:

# compile only (writes the SageMaker pipeline definition to a file)
cursus compile -d my_dag.json -c my_config.json -o pipeline.json

# validate DAG <-> config alignment without compiling
cursus compile -d my_dag.json -c my_config.json --validate-only

# compile, deploy to SageMaker, and start an execution
cursus compile -d my_dag.json -c my_config.json \
    --upsert --start --role arn:aws:iam::123456789012:role/MySageMakerRole

3. Build a DAG in Python

from cursus.api import PipelineDAG
from cursus.core import compile_dag_to_pipeline

# Nodes are step names; edges are data dependencies
dag = PipelineDAG()
for node in ["CradleDataLoading", "TabularPreprocessing", "XGBoostTraining"]:
    dag.add_node(node)
dag.add_edge("CradleDataLoading", "TabularPreprocessing")
dag.add_edge("TabularPreprocessing", "XGBoostTraining")

# config.json maps each node -> a config class (metadata.config_types)
pipeline = compile_dag_to_pipeline(dag=dag, config_path="config.json")

# Deploy / run when ready
pipeline.upsert(role_arn="arn:aws:iam::123456789012:role/MySageMakerRole")
pipeline.start()

See the Quickstart guide for the full walkthrough.

Key Features

  • 🎯 Graph-to-pipeline automation — a DAG of step names compiles straight to a SageMaker pipeline; the SageMaker step objects, wiring, and naming are generated for you.
  • 🧠 Intelligent dependency resolution — Cursus infers step connections and data flow by matching each step's declared outputs to the next step's declared inputs (semantic scoring), instead of hand-wiring properties paths.
  • 📄 Declarative, data-driven steps — every step is a single <step>.step.yaml interface unifying the script contract (I/O, env vars, job arguments) and the dependency spec; step builders are synthesized at runtime, with no hand-written builder classes to maintain.
  • 📦 A pre-built pipeline catalog — 44 ready-to-use DAGs across XGBoost, PyTorch, LightGBM, Bedrock and more, discoverable by framework, task type, and complexity.
  • 🧩 Extensible via step packs — define your own steps in a folder outside the installed package and Cursus discovers them as native, strictly additively (built-in steps are never removed).
  • 🛡️ Built-in validation — DAG↔config alignment, interface conformance, and dependency resolution are checked before you deploy (cursus validate, cursus alignment, --validate-only).
  • 🤖 Agent-ready (MCP) — a framework-neutral, self-documenting tool surface of 70 JSON-in/JSON-out tools across 12 namespaces mirrors the CLI/API for LLM agents. pip install "cursus[mcp]", then wire cursus-mcp into any MCP host (Claude Desktop, Cursor, Kiro). Read-only by default — state-changing tools are opt-in via env vars — with per-tool safety annotations. See the MCP server guide; cursus mcp help to inspect the tools.

How It Works

A DAG + config flows through layered subsystems to a SageMaker pipeline:

Subsystem Package Responsibility
DAG model cursus.api.dag PipelineDAG — nodes (step names) + edges (dependencies)
Compiler cursus.core.compiler PipelineDAGCompiler / compile_dag_to_pipeline — validate → resolve → build → assemble
Assembler cursus.core.assembler Turns resolved steps into a sagemaker Pipeline
Dependency resolver cursus.core.deps Matches producer outputs to consumer inputs (semantic scoring)
Step interfaces cursus.core.base + cursus.steps.interfaces Declarative <step>.step.yaml; builders synthesized at runtime
Registry & discovery cursus.registry + cursus.step_catalog Canonical step table, derived interface-first; step-pack discovery
Config system cursus.core.config_fields + cursus.api.factory Pydantic config classes; metadata.config_types node→class map
Pipeline catalog cursus.pipeline_catalog Pre-built shared DAGs + router (recommend_dag / load_shared_dag)
Validation cursus.validation Alignment / interface / dependency checks
Agent surface cursus.mcp The MCP tool surface
CLI cursus.cli 13 command groups

Read the Architecture & Design docs for the full picture.

What's Included

Count
Step types 57 registered (54 declarative .step.yaml interfaces) data loading, preprocessing, training, eval, calibration, registration, …
Pre-built DAGs 44 across 8 frameworks XGBoost, PyTorch, LightGBM, LightGBM-MT, XGBoost-MT, Bedrock, TSA, Dummy
CLI command groups 13 compile, dag, config, catalog, steps, strategies, pipeline-catalog, validate, alignment, registry, projects, exec-doc, mcp
MCP agent tools 70 across 12 namespaces discover, construct, validate, compile, author — for LLM agents

Command-Line Interface

cursus compile -d dag.json -c config.json -o pipeline.json   # DAG + config -> pipeline
cursus compile -d dag.json -c config.json --validate-only    # dry-run alignment report
cursus pipeline-catalog recommend --framework xgboost        # discover a pre-built DAG
cursus pipeline-catalog get-dag xgboost_complete_e2e         # inspect a catalog DAG
cursus catalog list                                          # browse available step types
cursus steps io XGBoostTraining                              # a step's declared I/O
cursus dag resolve TabularPreprocessing XGBoostTraining      # score dependency edges
cursus validate step-interface --all                         # validate every interface
cursus projects list                                         # discover pipeline projects
cursus mcp help                                              # explore the agent tool surface

Every group, subcommand, and flag is in the CLI reference.

Installation Options

Extra Installs For
(core) DAG model, compiler, catalog, CLI, MCP everything except heavy ML/data libs
processing pandas, numpy data-processing utilities & scripts
pytorch torch, lightning PyTorch training/eval steps
gbm xgboost, lightgbm gradient-boosting steps
nlp tokenizers, transformers text steps
jupyter notebook tooling interactive development
viz plotting libraries reports/visualizations
docs sphinx, furo, sphinx-click, … building this documentation
dev test/lint toolchain contributing
all pytorch + gbm + nlp + processing + jupyter + viz full runtime install 
pip install "cursus[all]"          # full ML runtime
pip install "cursus[dev]"          # contributor toolchain

📖 Documentation

🌐 Hosted Documentation → cursus.readthedocs.io

The full, auto-generated documentation site — rebuilt from the source on every release (also mirrored at tianpeiluke.github.io/cursus): Getting Started · Tutorials · Concepts & Architecture · How-to Guides · API Reference · CLI Reference · MCP Tools · Step Catalog · Pipeline Catalog

Design & developer notes (in-repo)

Who Should Use Cursus?

  • Data scientists & ML practitioners — go from a workflow sketch to a running SageMaker pipeline without writing SageMaker step glue; start from a catalog template and customize.
  • Platform & ML engineers — standardize pipeline construction, enforce DAG↔config alignment in CI, and extend the step library with your own step packs.
  • Organizations — a consistent, validated, reproducible path from graph to production pipeline, with less bespoke SageMaker code to maintain.

🤝 Contributing

Contributions are welcome! See the Developer Guide for:

Or author a step the guided way with the cursus mcp agent tools.

📄 License

Licensed under the MIT License — see LICENSE.

🔗 Links

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (2.9.35)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for LukeXie from gravatar.com LukeXie

Credits

Author/Maintainer: Tianpei Xie

License expression

MIT
View SPDX License List

Requires

Python >=3.9

Provides Extra

processing pytorch gbm nlp jupyter viz dev docs mcp all

Tags

sagemaker pipeline dag machine-learning aws automation mlops data-science workflow orchestration

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cursus-2.9.6.tar.gz (1.5 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

cursus-2.9.6-py3-none-any.whl (1.8 MB view details)

Uploaded Python 3

File details

Details for the file cursus-2.9.6.tar.gz.

File metadata

  • Download URL: cursus-2.9.6.tar.gz
  • Upload date:
  • Size: 1.5 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.12.7

File hashes

Hashes for cursus-2.9.6.tar.gz
Algorithm Hash digest
SHA256 6b9b75e86ce533d8ad63cf0bbd690e69713d4cf0dd9bcedabed0b548447e2d4b
MD5 55790e0d4108d4a9860d0098e2f3e7ed
BLAKE2b-256 232cfc523fa3644bb4fad8bb2d8d7138dfc72dc00bb51a65d5887999d79554a1

See more details on using hashes here.

File details

Details for the file cursus-2.9.6-py3-none-any.whl.

File metadata

  • Download URL: cursus-2.9.6-py3-none-any.whl
  • Upload date:
  • Size: 1.8 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.12.7

File hashes

Hashes for cursus-2.9.6-py3-none-any.whl
Algorithm Hash digest
SHA256 cf1d06a423fa7a6deea78af172e4dd021f9c324bb19cde1a0225bf08258dd0f2
MD5 bae2d2d62f5d995b7cc3a9b81c6603d2
BLAKE2b-256 2d662d44092d9af8467d745405f0d4babd06a8936fe6796b1b1e3a399ea34dae

See more details on using hashes here.

Release history Release notifications | RSS feed

2.9.35

2 files

2.9.31

2 files

2.9.28

2 files

2.9.27

2 files

2.9.26

2 files

2.9.17

2 files

2.9.13

2 files

2.9.12

2 files

This release

2.9.6 This release

2 files

2.9.2

2 files

2.9.1

2 files

2.9.0

2 files

2.8.5

2 files

2.8.4

2 files

2.8.2

2 files

2.8.1

2 files

2.8.0

2 files

2.6.0

2 files

2.5.10

2 files

2.5.8

2 files

2.5.4

2 files

2.3.0

2 files

2.2.1

2 files

2.2.0

2 files

2.1.0

2 files

2.0.0

2 files

1.10.0

2 files

1.9.0

2 files

1.8.1

2 files

1.8.0

2 files

1.7.1

2 files

1.7.0

2 files

1.6.0

2 files

1.5.3

2 files

1.5.2

2 files

1.5.1

2 files

1.5.0

2 files

1.4.7

2 files

1.4.6

2 files

1.4.5

2 files

1.4.4

2 files

1.4.3

2 files

1.4.2

2 files

1.4.1

2 files

1.4.0

2 files

1.3.9

2 files

1.3.8

2 files

1.3.7

2 files

1.3.6

2 files

1.3.5

2 files

1.3.4

2 files

1.3.3

2 files

1.3.2

2 files

1.3.1

2 files

1.3.0

2 files

1.2.6

2 files

1.2.5

2 files

1.2.4

2 files

1.2.3

2 files

1.2.2

2 files

1.2.1

2 files

1.2.0

2 files

1.1.1

2 files

1.1.0

2 files

1.0.12

2 files

1.0.11

2 files

1.0.10

2 files

1.0.9

2 files

1.0.8

2 files

1.0.7

2 files

1.0.6

2 files

1.0.5

2 files

1.0.4

2 files

1.0.3

2 files

1.0.2

2 files

1.0.1

1 file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page