AWS Glue Toolkit

A streamlined CLI utility designed to simplify the AWS Glue development lifecycle.

Disclaimer: This is an independent, community-maintained project. It is not affiliated with, endorsed by, or sponsored by Amazon Web Services (AWS). AWS, AWS Glue, and related marks are trademarks of Amazon.com, Inc. or its affiliates.

Installation

Requires Python 3.11+ and Docker (for all gtk commands; assumed installed, never installed by gtk). Installing the package adds the gtk command and a compatible pip release.

pip install aws-glue-toolkit

Example

A Glue job is a directory with a pyproject.toml and a source tree. Given this layout:

my-glue-job/
  pyproject.toml
  src/
    __main__.py
  tests/
    test_example.py

configure the job in pyproject.toml:

[project]
name = "my-glue-job"
version = "0.1.0"
dependencies = ["pandas>=2"]

[tool.aws-glue-toolkit]
glue_version = "5.1"
source = "src"
script = "__main__.py"
tests = "tests"

then run:

cd my-glue-job
gtk check .
gtk build .
gtk run .
gtk test .

See samples/README.md for a worked sample (Glue 5.0).

Configuration

Each job is a directory containing pyproject.toml. Unknown keys are ignored. The source directory and entry script must exist before gtk runs.

Pip configuration

For container pip (check, build, and dep install on run / test), gtk snapshots the host’s effective pip configuration (files + PIP_*, env wins) into a temporary pip.conf, mounts it, and sets PIP_CONFIG_FILE. Host-local keys (cache-dir, cert, target, and similar) are omitted.

# ~/.config/pip/pip.conf  or:
export PIP_EXTRA_INDEX_URL="https://my.company/simple"
gtk check .

Commands

[JOB-DIR] is the job directory path, passed as a positional argument or with --job-dir [JOB-DIR] (default: .). Docker must be available; the Glue image is pulled on first use. gtk never installs Docker.

For run / test: job dir mounted at /home/hadoop/workspace; non-empty project.dependencies install into an ephemeral PYTHONPATH target (Glue pins as constraints); stdio pass through; exit code is the container command’s (or pip’s if install fails). Same dependency forms as check / build.

check

Resolves project.dependencies in the Glue image for glue_version against bundled pins. Does not write files.

build

Writes a dependencies zip (host: .py under source) and a gluewheels zip (pip wheel in the Glue image). Path deps must be installable packages (pyproject.toml or setup.py); loose job modules belong under source. Both zips are always written; gluewheels omits packages already pinned on the image at the same version.

Dependency forms

Editable installs (-e) are rejected.

run

spark-submit on the entry script. Passes --JOB_NAME from project.name unless overridden. Extra --key value tokens after [JOB-DIR] go to getResolvedOptions (pass an explicit [JOB-DIR] when using .). Shuts down the Spark driver so the container returns.

test

python3 -m pytest. PYTHONPATH includes source (and the install target when deps are present). Default target is the configured tests dir; if the first forwarded token starts with -, that dir is still passed first (gtk test . -v); if it is a path, only those tokens go to pytest.

Exit codes

Success is 0; unhandled errors are 1. For run / test, a successful Docker launch returns the container command’s exit code (not limited to the table above).

Architecture

Two-layer module layout (core and shell) is documented in docs/architecture.md.

Contributing

Contributions are welcome. See CONTRIBUTING.md for setup and workflow. Participants are expected to follow the Code of Conduct. Report security vulnerabilities privately via SECURITY.md.

License

This project is licensed under the MIT License — see the LICENSE file for details.