AWS Glue Toolkit

A streamlined command-line utility designed to simplify the AWS Glue development lifecycle.

Disclaimer: This is an independent, community-maintained project. It is not affiliated with, endorsed by, or sponsored by Amazon Web Services (AWS). AWS, AWS Glue, and related marks are trademarks of Amazon.com, Inc. or its affiliates.

Installation

Requires Python 3.11+. Install Docker for run, test, and gtk build --mode container; the Glue image is pulled on first use. Default gtk build uses host pip (--mode host).

pip install aws-glue-toolkit

Example

A Glue job is a directory with a pyproject.toml and a source tree. Given this layout:

my-glue-job/
  pyproject.toml
  src/
    __main__.py
  tests/
    test_example.py

Configure the job in pyproject.toml:

[project]
name = "my-glue-job"
version = "0.1.0"
dependencies = ["pandas>=2"]

[tool.aws-glue-toolkit]
glue_version = "5.1"
source = "src"
script = "__main__.py"
tests = "tests"

Then run:

cd my-glue-job
gtk build .
gtk run .
gtk test .

Configuration

Each job is a directory containing pyproject.toml. gtk reads the [project] and [tool.aws-glue-toolkit] fields in the table below. Other keys in the file are skipped. The source directory and entry script must exist before gtk runs.

Commands

[JOB-DIR] is the job directory (default .) on build, run, and test.

Run gtk COMMAND --help for flags and behavior (for example gtk build --help).

For build --mode container, run, and test, gtk snapshots the host pip configuration for pip inside the container.

Build artifacts

Use gtk build to verify dependencies against Glue runtime pins. It writes two zips under the job directory:

Exit codes

Documented failure codes follow the Berkeley sysexits convention in the 64–78 band:

Success is 0; unhandled errors are 1. build uses the codes above. run and test return the container command’s exit code when pip install and the main command succeed (pytest exit code 1 means failing tests).

License

This project is licensed under the MIT License — see the LICENSE file for details.