gluekit

Introduction

gluekit is reusable AWS Glue job tooling for local-first development. It is designed for teams that want Glue job definitions, scripts, notebooks, and supporting artifacts in source control while still being able to pull from AWS, edit safely, validate locally, and push intentional changes back.

The command line is meant to feel closer to git and uv than to a cloud console: quick local commands, explicit artifact layouts, checkout-style context, and conversion workflows that smooth over AWS Glue quirks without hiding the underlying AWS behavior. A major motivation is making script and notebook work interchangeable enough for day-to-day development while still respecting that Glue scripts and Glue interactive notebooks are not identical runtime surfaces.

Local Development Runtimes: gluekit uses SQLFrame by default for fast local runs and can retain a real installed PySpark engine with gluekit run --runtime pyspark. The scoped awsglue compatibility shim remains available in either mode. See the Local Development documentation for details.

Suggested workflows

For AWS CLI-oriented setup, sync, and deployment recipes, start with the AWS CLI recipes folder, especially docs/aws_cli_recipes/workflows.md.

Common starting points:

• Pull a real Glue job into source control, manage local job properties and runtime parameters, and push selected artifacts back with gluekit pull, gluekit checkout, gluekit job, gluekit params, and gluekit push.
• Use gluekit local setup, gluekit local s3, gluekit local ssm, and gluekit run to build a repeatable local test fixture before touching AWS.
• Keep account-specific values such as IAM roles, temp buckets, and script buckets in profile-scoped saved params so local configs can move between dev/prod-like AWS profiles without hand-editing generated artifacts.

Script-based development with notebook interoperability

gluekit supports a script-first workflow that can still interoperate with notebooks. Python scripts can use #%% cell tags to mark notebook-like cells. These tags divide a .py file into executable notebook cells; Markdown cells can use #%% [markdown]. This follows the same family of conventions described by VS Code's Python Interactive/Jupyter documentation, where Python files can contain Jupyter-style cell delimiters and be converted back into notebooks.

The intent is flexibility: you can develop in a normal Python file, use editor support for cell-by-cell execution, keep clean script diffs in git, and generate a Glue notebook when that is the deployment or debugging surface you need.

If you have already selected the job with checkout:

gluekit checkout {jobname}
gluekit script --to-notebook

If you have not checked out the job, pass it explicitly:

gluekit script {jobname} --to-notebook

Gluekit vs AWS CLI terminology

Some gluekit commands look like AWS CLI commands, but the scope is different:

• gluekit pull, gluekit push, gluekit build --push, and gluekit init use real AWS Glue/S3 APIs. On those commands, --profile means an AWS CLI credential profile, the same profile name boto3 would use.
• gluekit checkout --profile and gluekit config set --profile update local gluekit state in .gluekit/glue_set.json. The value is a gluekit profile scope for saved params; it usually matches an AWS profile name, but those commands do not contact AWS.
• gluekit local ... and gluekit run are local test tooling. Their S3, SSM, region, job, and profile values are labels or mocked fixture values stored in JSON and used by local moto/boto3 mocks. They do not create, read, or delete real AWS resources. Install the job-dev optional dependency to use these local development features.

Install

Recommended

pip install "gluekit[job-dev]"

This installs the CLI plus the local job development extras used by gluekit run and gluekit local, which is the expected setup for most users.

Install gluekit[spark-dev] instead when tests require a real local Spark engine. Projects should additionally pin the PySpark version matching their AWS Glue runtime.

Using uv

Ensure you have uv installed globally (see uv installation docs).

Install as a global command line tool:

uv tool install "gluekit[job-dev]"

Or add it to a local job development project:

uv add "gluekit[job-dev]" --dev

Lightweight install

If you only need Glue job management commands and do not plan to run jobs locally:

pip install gluekit

Quick start

1. Install the AWS command line tool and configure your credentials. If using sso, need to log in with

aws sso login --profile your-profile-name

2. Right now, the best way to get started is to check out the Command line help documentation below, which is auto-generated from the live CLI --help output and includes usage examples for each command.

Quick start commands

Glue job management

Initialize a workspace from your AWS CLI credential profiles:

gluekit init

Pull or check out the job you want to work on, then make focused config changes:

gluekit checkout my-job --profile survey-dev
gluekit job --role arn:aws:iam::111111111111:role/glue-dev
gluekit params --temp-dir s3://dev-bucket/tmp/

Add local artifacts or shared files to the checked-out job:

gluekit params --add-additional-python-modules dist/my_pkg-0.0.1-py3-none-any.whl
gluekit params --add-additional-python-files glue/lib/helpers.py

Convert between Glue scripts and notebooks when your workflow changes:

gluekit script my-job --to-notebook
gluekit notebook my-job --to-script

Build Python artifacts and optionally push the checked-out job updates:

gluekit build
gluekit build --bump-version patch --package-whl --push

Local Glue job development

Create and check out the local setup with mocked AWS resource mappings:

gluekit local setup \
  --profile local-dev \
  --s3-root sources/s3 \
  --s3-bucket local-dev=tests/fixtures \
  --ssm-param /app/runtime=local

Inspect the saved setup, add SSM parameters, and run the Glue script locally:

gluekit local status
gluekit local ssm put /app/runtime local
gluekit run glue/scripts/my-job.py --glue-arg run_date=2026-06-18
gluekit run glue/scripts/my-job.py --runtime pyspark --spark-master 'local[2]'

Use --s3-root sources/s3 to map an entire mocked profile/account S3 namespace, where sources/s3/{bucket}/{prefix/key} appears as s3://{bucket}/{prefix/key}. Use --s3-bucket as {s3 bucket in mocked profile}={local directory} for a single bucket, and --ssm-param as {ssm parameter in mocked profile}={local parameter value}.

Local setup writes fixture data to .gluekit/local.json and records the active local settings under checkout.local in .gluekit/glue_set.json. The local --profile value is saved in both places as a cross-reference label for comparison and validation; gluekit local setup does not authenticate to AWS.

Generated metadata

gluekit now includes two checked-in metadata artifacts used by notebook conversion and Glue job validation helpers:

• src/gluekit/resources/glue_job_schema.json is generated from the local botocore AWS Glue service model.
• src/gluekit/resources/magic_map.json is a curated mapping of Glue job config fields to AWS Glue Interactive Sessions magics.

The schema artifact is generated by:

uv run python scripts/extract_glue_schema.py

Rerun that script when changing schema-driven validation logic, when upgrading boto3/botocore in a way that could change the Glue service model, or when you need to refresh the checked-in schema after AWS Glue API changes.

The magic map is intentionally maintained by hand because it reflects documented notebook behavior rather than raw AWS API structure. When updating notebook conversion behavior, review the AWS Glue Interactive Sessions magics docs and keep src/gluekit/resources/magic_map.json in sync.

Resource-oriented configuration workflow

The resource commands are checkout-first. Select a job once with gluekit checkout my-job, then use gluekit job for AWS Glue job-definition properties and gluekit params for DefaultArguments and NonOverridableArguments.

Use gluekit notebook --sync-params or gluekit script --sync-params to refresh generated metadata from local config. Reverse imports are explicit through --to-config, and representation conversion is directional through gluekit notebook --to-script and gluekit script --to-notebook.

Reverse imports treat the artifact as authoritative only for fields represented by Glue interactive-session magics and DefaultArguments: changed values are updated, represented values removed from the artifact are removed from local config, and unrelated job properties are preserved. Use --dry-run to inspect those changes before writing.

The deprecated edit, convert, sync, add, and update commands remain compatibility wrappers and emit migration guidance.

To target a different local config for one command, pass the job name directly to the resource command, such as gluekit job <job-name> ... or gluekit params <job-name> ....

Profile-specific params and build automation

Use profile-scoped saved params for account-dependent values such as IAM roles, script buckets, temp buckets, and other Glue default arguments. These are local gluekit param scopes saved in .gluekit/glue_set.json; they are only used as real AWS credential profiles by commands that call AWS, such as pull, push, build --push, and init.

gluekit config set my-job --profile survey-dev --Role=arn:aws:iam::111111111111:role/glue-dev
gluekit config set my-job --profile survey-prod --Role=arn:aws:iam::222222222222:role/glue-prod
gluekit config set my-job --profile survey-prod --command_script_location=s3://prod-bucket/scripts/my-job.py

Then check out the job and gluekit profile scope together:

gluekit checkout my-job --profile survey-prod

After a job is checked out, switch only the active profile scope with:

gluekit checkout --profile survey-dev

Checkout stores the active gluekit profile scope in .gluekit/glue_set.json and applies the matching saved params to the local config immediately, including mapped Glue config paths such as Role, Command.ScriptLocation, DefaultArguments.*, and SourceControlDetails.*. gluekit pull validates existing active profile mappings against the pulled config; if the active profile has no mappings yet but another profile exists for the job, pull detects and saves role plus S3/config mappings using the same profile comparison rules as gluekit init. gluekit push validates that the active profile mappings already align with the local config before uploading, and reports every mismatch instead of rewriting the config at push time. gluekit pull, gluekit push, and gluekit build --push also use the active profile value as the AWS CLI credential profile when --profile is not passed.

The common package release loop can now be a single command:

gluekit build --bump-version patch --package-whl --push

That bumps pyproject.toml, builds artifacts, refreshes the checked-out job's tracked dist/*.whl in --additional-python-modules, applies active profile params before deriving S3 paths, and pushes the selected Glue artifacts/config. Use --dry-run to preview the build, wheel update, and push steps.

For first-time repository setup, gluekit init scans your AWS CLI config profiles, prompts for the Glue job to inspect in each selected profile, can save profile-specific role and S3 params locally, and sets the first checked-out job/profile scope.

Command line help documentation

Auto-generated from live CLI --help output.

gluekit --help

Usage: gluekit [OPTIONS] COMMAND [ARGS]...

Options:
  --install-completion  Install completion for the current shell.
  --show-completion     Show completion for the current shell, to copy it or
                        customize the installation.
  --help                Show this message and exit.

Commands:
  build     Build local artifacts; only touches AWS when --push is used.
  checkout  Replace the active local checkout selection.
  edit      Edit one local Glue job config with explicit mutation flags.
            (DEPRECATED)
  clone     Clone local Glue job configurations and artifacts.
  set       Set reusable local key/value parameters for a Glue job.
  convert   Convert Glue jobs between script and notebook formats. (DEPRECATED)
  init      Initialize local repo-level Gluekit settings from AWS CLI...
  list      List real AWS Glue jobs in the configured AWS account.
  notebook  Operate on the local AWS Glue interactive-session notebook.
  params    Manage local AWS Glue runtime and job arguments.
  pull      Download real AWS Glue job artifacts and config into the local...
  push      Upload local artifacts to real AWS Glue/S3 and optionally...
  run       Run a Glue script locally with emulated Glue libraries and...
  remove    Remove local Glue job configs and local artifacts.
  status    Show the active local checkout and its local-to-S3 artifact...
  sync      Sync generated Glue config metadata cells in local... (DEPRECATED)
  script    Operate on the local AWS Glue Python script.
  job       Manage local AWS Glue job-definition properties.
  add       Compatibility command for inferred artifact and module additions.
  update    Compatibility command for generic property updates.
  config    Show and set local reusable Glue config parameters.
  local     Manage local-only Glue development setups and mocked AWS fixtures.

gluekit pull --help

Usage: gluekit pull [OPTIONS] [JOB_NAME] [TARGET_JOB]

  Download real AWS Glue job artifacts and config into the local workspace.

Arguments:
  [JOB_NAME]    Glue job name or pattern to pull. Use "*" for all jobs.
  [TARGET_JOB]  Optional local job name to write pulled artifacts to.

Options:
  -f, --force                     Download even if the local config is newer or
                                  equal to LastModifiedOn.
  --dry-run                       Show what would be downloaded without writing
                                  files.
  -i, --include TEXT              Include only specific components (config,
                                  script, notebook, extra-py-files, extra-
                                  files).
  -x, --exclude TEXT              Exclude specific components (config, script,
                                  notebook, extra-py-files, extra-files).
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  --write-job-list / --no-write-job-list
                                  Write glue/glue_full_job_list.csv during the
                                  run.  [default: write-job-list]
  -p, --profile TEXT              AWS CLI credential profile used for real AWS
                                  Glue/S3 API calls; not a gluekit local test
                                  profile.
  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
                                  sso login' when credentials are missing or
                                  expired.  [default: auto-login]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit pull my-job --profile my-sso-profile`

  - `gluekit pull source-job target-job --profile my-sso-profile`

  - `gluekit pull "my-job-\*" --force --include config,script --exclude
  notebook`

  - `gluekit pull "\*" --dry-run --profile my-sso-profile --no-auto-login`

gluekit job --help

Usage: gluekit job [OPTIONS] [JOB_NAME]

  Manage local AWS Glue job-definition properties.

Arguments:
  [JOB_NAME]  Glue job name. Defaults to the active checkout selection.

Options:
  --description TEXT              Set the job description.
  --role TEXT                     Set the IAM role for the job.
  --glue-version TEXT             Set the Glue version.
  --worker-type TEXT              Set the worker type.
  --number-of-workers INTEGER     Set the number of workers.
  --timeout INTEGER               Set the job timeout in minutes.
  --max-retries INTEGER           Set the max retry count.
  --execution-class TEXT          Set the execution class.
  --max-concurrent-runs INTEGER   Set the max concurrent runs.
  --job-run-queuing-enabled / --no-job-run-queuing-enabled
                                  Enable or disable job run queuing.
  --security-configuration TEXT   Set the security configuration name.
  --maintenance-window TEXT       Set the maintenance window.
  --log-uri TEXT                  Set the log URI.
  --command-name TEXT             Set Command.Name.
  --python-version TEXT           Set Command.PythonVersion.
  --runtime TEXT                  Set Command.Runtime.
  --remote-script-location TEXT   Set Command.ScriptLocation.
  --local-script-location TEXT    Set SourceControlDetails.ScriptLocation.
  --notebook-location TEXT        Set SourceControlDetails.NotebookLocation and
                                  NotebookPath.
  --source-control-provider TEXT  Set SourceControlDetails.Provider.
  --source-control-owner TEXT     Set SourceControlDetails.Owner.
  --source-control-repository TEXT
                                  Set SourceControlDetails.Repository.
  --source-control-branch TEXT    Set SourceControlDetails.Branch.
  --source-control-folder TEXT    Set SourceControlDetails.Folder.
  --add-connection TEXT           Add one or more Glue connections.
  --remove-connection TEXT        Remove one or more Glue connections.
  --dry-run                       Show what would be updated without writing
                                  files.
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit job --description 'Updated job description' --glue-version 5.0`

  - `gluekit job my-job --worker-type G.1X --number-of-workers 5`

gluekit params --help

Usage: gluekit params [OPTIONS] [JOB_NAME]

  Manage local AWS Glue runtime and job arguments.

Arguments:
  [JOB_NAME]  Glue job name. Defaults to the active checkout selection.

Options:
  --add-additional-python-modules TEXT
                                  Add one or more values to --additional-python-
                                  modules.
  --remove-additional-python-modules TEXT
                                  Remove one or more values from --additional-
                                  python-modules.
  --add-extra-py-files TEXT       Track local relative Python files in
                                  SourceControlDetails.ExtraPyFiles and --extra-
                                  py-files.
  --remove-extra-py-files TEXT    Remove local Python files from
                                  SourceControlDetails.ExtraPyFiles and --extra-
                                  py-files.
  --add-extra-jars TEXT           Add one or more values to --extra-jars.
  --remove-extra-jars TEXT        Remove one or more values from --extra-jars.
  --add-additional-python-files TEXT
                                  Track local paths in
                                  SourceControlDetails.AdditionalPythonFiles and
                                  --extra-py-files.
  --remove-additional-python-files TEXT
                                  Remove local paths from
                                  SourceControlDetails.AdditionalPythonFiles and
                                  --extra-py-files.
  --add-extra-files TEXT          Track local paths in
                                  SourceControlDetails.ExtraFiles and --extra-
                                  files.
  --remove-extra-files TEXT       Remove local paths from
                                  SourceControlDetails.ExtraFiles and --extra-
                                  files.
  --add-default-argument TEXT     Set a DefaultArguments entry using KEY=VALUE.
                                  Repeat as needed.
  --remove-default-argument TEXT  Remove a DefaultArguments entry by key. Repeat
                                  as needed.
  --add-non-overridable-argument TEXT
                                  Set a NonOverridableArguments entry using
                                  KEY=VALUE. Repeat as needed.
  --remove-non-overridable-argument TEXT
                                  Remove a NonOverridableArguments entry by key.
                                  Repeat as needed.
  --enable-metrics / --disable-metrics
                                  Set DefaultArguments.--enable-metrics to true
                                  or false.
  --enable-observability-metrics / --disable-observability-metrics
                                  Set DefaultArguments.--enable-observability-
                                  metrics to true or false.
  --enable-glue-datacatalog / --disable-glue-datacatalog
                                  Set DefaultArguments.--enable-glue-datacatalog
                                  to true or false.
  --enable-continuous-cloudwatch-log / --disable-continuous-cloudwatch-log
                                  Set DefaultArguments.--enable-continuous-
                                  cloudwatch-log to true or false.
  --enable-spark-ui / --disable-spark-ui
                                  Set DefaultArguments.--enable-spark-ui to true
                                  or false.
  --enable-job-insights / --disable-job-insights
                                  Set DefaultArguments.--enable-job-insights to
                                  true or false.
  --enable-auto-scaling / --disable-auto-scaling
                                  Set DefaultArguments.--enable-auto-scaling to
                                  true or false.
  --job-bookmark-option TEXT      Set DefaultArguments.--job-bookmark-option.
  --temp-dir TEXT                 Set DefaultArguments.--TempDir.
  --spark-event-logs-path TEXT    Set DefaultArguments.--spark-event-logs-path.
  --datalake-formats TEXT         Set DefaultArguments.--datalake-formats.
  --spark-conf TEXT               Set DefaultArguments.--conf.
  --continuous-log-log-group TEXT
                                  Set DefaultArguments.--continuous-log-
                                  logGroup.
  --continuous-log-log-stream-prefix TEXT
                                  Set DefaultArguments.--continuous-log-
                                  logStreamPrefix.
  --continuous-log-conversion-pattern TEXT
                                  Set DefaultArguments.--continuous-log-
                                  conversionPattern.
  --executor-cores INTEGER        Set DefaultArguments.--executor-cores.
  --customer-driver-env-vars TEXT
                                  Set DefaultArguments.--customer-driver-env-
                                  vars.
  --customer-executor-env-vars TEXT
                                  Set DefaultArguments.--customer-executor-env-
                                  vars.
  --python-modules-installer-option TEXT
                                  Set DefaultArguments.--python-modules-
                                  installer-option.
  --dry-run                       Show what would be updated without writing
                                  files.
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit params --temp-dir s3://bucket/tmp --enable-metrics`

  - `gluekit params my-job --add-default-argument survey_id=test`

gluekit notebook --help

Usage: gluekit notebook [OPTIONS] [JOB_NAME]

  Operate on the local AWS Glue interactive-session notebook.

Arguments:
  [JOB_NAME]  Glue job name. Defaults to the active checkout selection.

Options:
  --sync-params, --add-params   Refresh Gluekit-managed interactive-session
                                magic cells.
  --to-script                   Convert the notebook to a Glue script.
  --to-config                   Import represented magics into local job config.
  --use-config / --script-only  Use config metadata and configured paths during
                                conversion.  [default: use-config]
  --dry-run                     Show changes without writing local files.
  --config-dir PATH             [default: glue/configs]
  --help                        Show this message and exit.

  **Examples**

  - `gluekit notebook --sync-params`

  - `gluekit notebook --to-script`

  - `gluekit notebook --to-config --dry-run`

gluekit script --help

Usage: gluekit script [OPTIONS] [JOB_NAME]

  Operate on the local AWS Glue Python script.

Arguments:
  [JOB_NAME]  Glue job name. Defaults to the active checkout selection.

Options:
  --sync-params, --add-params   Refresh Gluekit-managed configuration metadata.
  --to-notebook                 Convert the script to a Glue notebook.
  --to-config                   Import Gluekit-generated metadata into local job
                                config.
  --use-config / --script-only  Use config metadata and configured paths during
                                conversion.  [default: use-config]
  --dry-run                     Show changes without writing local files.
  --config-dir PATH             [default: glue/configs]
  --help                        Show this message and exit.

  **Examples**

  - `gluekit script --sync-params`

  - `gluekit script --to-notebook`

  - `gluekit script --to-config --dry-run`

gluekit status --help

Usage: gluekit status [OPTIONS]

  Show the active local checkout and its local-to-S3 artifact mappings.

Options:
  --config-dir PATH  Directory containing Glue job config files.  [default:
                     glue/configs]
  --help             Show this message and exit.

  **Examples**

  - `gluekit status`

  - `gluekit status --config-dir glue/configs`

gluekit run --help

Usage: gluekit run [OPTIONS] SCRIPT_PATH

  Run a Glue script locally with emulated Glue libraries and mocked AWS
  services.

Arguments:
  SCRIPT_PATH  Path to the Glue Python script to execute locally.  [required]

Options:
  --glue-arg TEXT       Glue script argument in KEY=VALUE form; passed through
                        as --KEY VALUE.
  --job-name TEXT       Default JOB_NAME used when not provided in script args.
                        [default: local-glue-job]
  --create-bucket TEXT  Create a mocked S3 bucket only inside the local run.
                        Repeat as needed.
  --ssm-parameter TEXT  Seed a mocked SSM parameter only inside the local run as
                        NAME=VALUE. Repeat as needed.
  --aws-region TEXT     Mock AWS region used for local boto3 clients. Defaults
                        to us-east-1.
  --config-file FILE    Local fixture config file for mocked S3/SSM. Defaults to
                        .gluekit/local.json when local settings are checked out.
  --runtime TEXT        Local DataFrame runtime: sqlframe (fast default) or
                        pyspark (real local Spark).  [default: sqlframe]
  --spark-master TEXT   Spark master URL used when --runtime=pyspark.  [default:
                        local[2]]
  --help                Show this message and exit.

  **Examples**

  - `gluekit run ../ppsc-survey/glue/scripts/job_a.py --create-bucket my-input-
  bucket`

  - `gluekit run glue/scripts/job-a.py --ssm-parameter /app/env=dev --glue-arg
  report_date=2026-06-18`

  - `gluekit run glue/scripts/job-a.py -- --JOB_NAME custom-local-job
  --report_date 2026-06-18`

gluekit local --help

Usage: gluekit local [OPTIONS] COMMAND [ARGS]...

  Manage local-only Glue development setups and mocked AWS fixtures.

Options:
  --help  Show this message and exit.

Commands:
  setup   Create or update and check out the local-only development setup.
  status  Show the active local development setup.
  s3      Manage local mocked S3 fixture mappings; does not call AWS S3.
  ssm     Manage local mocked SSM Parameter Store values; does not call AWS...

gluekit local setup --help

Usage: gluekit local setup [OPTIONS]

  Create or update and check out the local-only development setup.

Options:
  --profile TEXT    Optional gluekit/AWS profile name to associate with this
                    local setup in checkout state and .gluekit/local.json; this
                    command does not authenticate to AWS.
  --s3-bucket TEXT  Map a mocked S3 bucket to a local directory in {s3 bucket in
                    mocked profile}={local directory} form. Repeat as needed.
  --s3-root PATH    Map a local directory as the full mocked S3 namespace for
                    this setup/profile. Expected layout: {local s3
                    root}/{bucket}/{prefix/key}.
  --ssm-param TEXT  Map a mocked SSM parameter to a local value in {ssm
                    parameter in mocked profile}={local parameter value} form.
                    Repeat as needed.
  --help            Show this message and exit.

  **Examples**

  - `gluekit local setup --profile local-dev --s3-root sources/s3`

  - `gluekit local setup --s3-bucket local-dev=tests/fixtures`

  - `gluekit local setup --ssm-param /app/runtime=local`

gluekit local status --help

Usage: gluekit local status [OPTIONS]

  Show the active local development setup.

Options:
  --help  Show this message and exit.

  **Examples**

  - `gluekit local status`

gluekit local s3 --help

Usage: gluekit local s3 [OPTIONS] COMMAND [ARGS]...

  Manage local mocked S3 fixture mappings; does not call AWS S3.

Options:
  --help  Show this message and exit.

Commands:
  cp       Copy a local file or mocked S3 object in the fixture config;...
  mount    Map a local directory recursively into mocked S3 keys; does not...
  unmount  Remove a recursive local-directory-to-mocked-S3 mapping.
  mv       Move a local file or mocked S3 object in the fixture config;...
  rm       Remove mapped mocked S3 objects from the local fixture config.
  ls       List mocked S3 buckets inferred from mappings, or mapped objects.

gluekit local s3 cp --help

Usage: gluekit local s3 cp [OPTIONS] SOURCE DESTINATION

  Copy a local file or mocked S3 object in the fixture config; does not call
  AWS.

Arguments:
  SOURCE       Local path or mocked S3 URI source.  [required]
  DESTINATION  Local path or mocked S3 URI destination.  [required]

Options:
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 cp tests/fixtures/input.json s3://my-bucket/input.json`

  - `gluekit local s3 cp s3://my-bucket/input.json s3://my-bucket/copy.json`

  - `gluekit local s3 cp s3://my-bucket/input.json tmp/input.json`

gluekit local s3 mount --help

Usage: gluekit local s3 mount [OPTIONS] SOURCE_DIR DESTINATION

  Map a local directory recursively into mocked S3 keys; does not call AWS.

Arguments:
  SOURCE_DIR   Local directory to map recursively.  [required]
  DESTINATION  Mocked S3 bucket or prefix base for mapped files.  [required]

Options:
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 mount tests/fixtures s3://local-dev/input/`

  - `gluekit local s3 mount data/mock s3://bucket-a`

gluekit local s3 unmount --help

Usage: gluekit local s3 unmount [OPTIONS] TARGET

  Remove a recursive local-directory-to-mocked-S3 mapping.

Arguments:
  TARGET  Local source directory or mocked S3 destination to remove.  [required]

Options:
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 unmount tests/fixtures`

  - `gluekit local s3 unmount s3://local-dev/input/`

gluekit local s3 mv --help

Usage: gluekit local s3 mv [OPTIONS] SOURCE DESTINATION

  Move a local file or mocked S3 object in the fixture config; does not call
  AWS.

Arguments:
  SOURCE       Local path or mocked S3 URI source.  [required]
  DESTINATION  Local path or mocked S3 URI destination.  [required]

Options:
  --delete-local      Delete the local source file after mapping a LocalPath to
                      S3.
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 mv s3://my-bucket/input.json s3://my-bucket/renamed.json`

  - `gluekit local s3 mv s3://my-bucket/input.json tmp/input.json`

gluekit local s3 rm --help

Usage: gluekit local s3 rm [OPTIONS] URI

  Remove mapped mocked S3 objects from the local fixture config.

Arguments:
  URI  Mapped mocked S3 object or prefix to remove.  [required]

Options:
  -r, --recursive     Remove all mapped objects under the S3 prefix.
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 rm s3://my-bucket/input.json`

  - `gluekit local s3 rm s3://my-bucket/prefix/ --recursive`

gluekit local ssm --help

Usage: gluekit local ssm [OPTIONS] COMMAND [ARGS]...

  Manage local mocked SSM Parameter Store values; does not call AWS SSM.

Options:
  --help  Show this message and exit.

Commands:
  put  Create or update a mocked SSM parameter with local-only syntax.
  get  Show one mocked SSM parameter with local-only syntax.
  rm   Delete one mocked SSM parameter with local-only syntax.
  ls   List mocked SSM parameters with local-only syntax.

gluekit local ssm put --help

Usage: gluekit local ssm put [OPTIONS] NAME VALUE

  Create or update a mocked SSM parameter with local-only syntax.

Arguments:
  NAME   Mocked SSM parameter name.  [required]
  VALUE  Mocked SSM parameter value.  [required]

Options:
  --overwrite / --no-overwrite  Overwrite an existing parameter.  [default:
                                overwrite]
  --config-file PATH            Local fixture config file to update. Defaults to
                                .gluekit/local.json.
  --help                        Show this message and exit.

  **Examples**

  - `gluekit local ssm put /app/env dev`

gluekit build --help

Usage: gluekit build [OPTIONS]

  Build local artifacts; only touches AWS when --push is used.

Options:
  --job-name TEXT                 Checked-out or explicit local Glue job to
                                  update when using --package-whl or --push.
  --dry-run                       Show what would be built and zipped without
                                  running build commands.
  --bump-version, --bump TEXT     Bump pyproject.toml [project].version before
                                  building: major, minor, patch, or X.Y.Z.
  --build-tool TEXT               Build frontend to use: auto, uv, or build.
                                  [default: auto]
  --out-dir PATH                  The output directory to which distributions
                                  should be written.
  --no-version-suffix             Remove the version suffix from generated zip
                                  artifact names.
  -v, --verbose                   Print build commands before execution.
  --package-whl                   After building, replace the tracked dist/*.whl
                                  in the selected Glue config.
  --push                          Push the selected Glue config and artifacts
                                  after building.
  -i, --include TEXT              For --push, include only specific components
                                  (script, notebook, additional-python-modules,
                                  extra-files, job-config).
  -x, --exclude TEXT              For --push, exclude specific components
                                  (script, notebook, additional-python-modules,
                                  extra-files, job-config).
  --update-config / --no-update-config
                                  For --push, update Glue job configuration
                                  after uploading files.  [default: update-
                                  config]
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  -p, --profile TEXT              AWS CLI credential profile for profile-scoped
                                  config params and, with --push, real AWS
                                  Glue/S3 API calls.
  --auto-login / --no-auto-login  For --push with a real AWS profile,
                                  automatically run 'aws sso login' when
                                  credentials are missing or expired.  [default:
                                  auto-login]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit build`

  - `gluekit build --build-tool uv`

  - `gluekit build --out-dir build/artifacts --no-version-suffix`

  - `gluekit build --build-tool build --dry-run`

gluekit push --help

Usage: gluekit push [OPTIONS] [JOB_NAME] [TARGET_JOB]

  Upload local artifacts to real AWS Glue/S3 and optionally update the job
  config.

Arguments:
  [JOB_NAME]    Glue job name or pattern to push. Use "*" for all configs.
  [TARGET_JOB]  Optional remote Glue job name to push to.

Options:
  --dry-run                       Show what would be uploaded without writing
                                  files.
  -i, --include TEXT              Include only specific components (script,
                                  notebook, additional-python-modules, extra-
                                  files, job-config).
  -x, --exclude TEXT              Exclude specific components (script, notebook,
                                  additional-python-modules, extra-files, job-
                                  config).
  --update-config / --no-update-config
                                  Update Glue job configuration after uploading
                                  files.  [default: update-config]
  --build                         Build local package artifacts in the workspace
                                  root before push.
  --build-tool TEXT               Build frontend for --build: auto, uv, or
                                  build.  [default: auto]
  -v, --verbose                   Print local build command(s) before execution.
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  -p, --profile TEXT              AWS CLI credential profile used for real AWS
                                  Glue/S3 API calls; not a gluekit local test
                                  profile.
  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
                                  sso login' when credentials are missing or
                                  expired.  [default: auto-login]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit push my-job --profile my-sso-profile`

  - `gluekit push source-job target-job --profile my-sso-profile`

  - `gluekit push "my-job-\*" --include script,job-config --exclude notebook`

  - `gluekit push my-job --build --build-tool auto --dry-run`

gluekit set --help

Usage: gluekit set [OPTIONS] [JOB_NAME]

  Set reusable local key/value parameters for a Glue job.

Arguments:
  [JOB_NAME]  Glue job name to save params for.

Options:
  --global            Save params as global defaults for all jobs.
  -p, --profile TEXT  Save params under this gluekit profile scope, usually
                      matching an AWS CLI profile name; does not contact AWS.
  --help              Show this message and exit.

  **Examples**

  - `gluekit set my-job --script_location glue/scripts/my-job.py --extra_files
  s3://my-bucket/shared/config.json`

  - `gluekit pull my-job --profile my-sso-profile`

  - `gluekit set --global --additional_python_modules "pydantic==2.11.7"`

  - `gluekit push "my-job-\*" --profile my-sso-profile`

  - `gluekit config set my-job --default_arguments.--TempDir s3://my-
  bucket/tmp/`

  - `gluekit config get my-job`

  - `gluekit config get`

  - `gluekit set my-job --extra_py_files s3://my-
  bucket/dist/gluekit-0.0.1-py3-none-any.whl`

  - `gluekit push my-job --build --build-tool auto --profile my-sso-profile`

gluekit config --help

Usage: gluekit config [OPTIONS] COMMAND [ARGS]...

  Show and set local reusable Glue config parameters.

Options:
  --help  Show this message and exit.

Commands:
  set  Set reusable local Glue config params by job or globally.
  get  Show stored local Glue config params.

gluekit config set --help

Usage: gluekit config set [OPTIONS] [JOB_NAME]

  Set reusable local Glue config params by job or globally.

Arguments:
  [JOB_NAME]  Glue job name to save params for.

Options:
  --global            Save params as global defaults for all jobs.
  -p, --profile TEXT  Save params under this gluekit profile scope, usually
                      matching an AWS CLI profile name; does not contact AWS.
  --help              Show this message and exit.

  **Examples**

  - `gluekit config set my-job --default_arguments.--TempDir s3://my-
  bucket/tmp/`

  - `gluekit config set --global --additional_python_modules "pydantic==2.11.7"`

gluekit config get --help

Usage: gluekit config get [OPTIONS] [JOB_NAME]

  Show stored local Glue config params.

Arguments:
  [JOB_NAME]  Optional Glue job name to view merged params (global + job).

Options:
  -p, --profile TEXT  Include params from this gluekit profile scope, usually
                      matching an AWS CLI profile name; does not contact AWS.
  --help              Show this message and exit.

  **Examples**

  - `gluekit config get my-job`

  - `gluekit config get`

gluekit list --help

Usage: gluekit list [OPTIONS]

  List real AWS Glue jobs in the configured AWS account.

Options:
  --help  Show this message and exit.

  **Examples**

  - `gluekit list --help`

  - `gluekit init --help`

gluekit init --help

Usage: gluekit init [OPTIONS]

  Initialize local repo-level Gluekit settings from AWS CLI profiles.

Options:
  --aws-config PATH               Local AWS CLI config file to scan for
                                  credential profile names; defaults to
                                  AWS_CONFIG_FILE or ~/.aws/config.
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  --auto-login / --no-auto-login  For selected real AWS profiles, automatically
                                  run 'aws sso login' when credentials are
                                  missing or expired.  [default: auto-login]
  --detect-roles / --no-detect-roles
                                  Fetch the selected real AWS Glue job in each
                                  profile and save its Role as local gluekit
                                  params.  [default: detect-roles]
  --detect-buckets / --no-detect-buckets
                                  Compare S3 URIs in real AWS Glue job configs
                                  across profiles and save local gluekit profile
                                  mappings.  [default: detect-buckets]
  --match-threshold INTEGER       Minimum rapidfuzz score for non-exact S3 key
                                  matches.  [default: 85]
  --baseline-profile TEXT         Selected AWS CLI profile to store as the first
                                  gluekit checkout scope and S3 comparison
                                  baseline.
  --profile-job TEXT              Real AWS Glue job to inspect for an AWS CLI
                                  profile, as PROFILE=GLUE_JOB_NAME. May be
                                  repeated.
  --help                          Show this message and exit.

  **Examples**

  - `gluekit init`

Getting started with your project

1. Create a New Repository

First, create a repository on GitHub with the same name as this project, and then run the following commands:

git init -b main
git add .
git commit -m "init commit"
git remote add origin git@github.com:mbkranz/gluekit.git
git push -u origin main

2. Set Up Your Development Environment

Then, install the environment and the pre-commit hooks with

make install

This will also generate your uv.lock file

3. Run the pre-commit hooks

Initially, the CI/CD pipeline might be failing due to formatting issues. To resolve those run:

uv run pre-commit run -a

4. Commit the changes

Lastly, commit the changes made by the two steps above to your repository.

git add .
git commit -m 'Fix formatting issues'
git push origin main

You are now ready to start development on your project! The CI/CD pipeline will be triggered when you open a pull request, merge to main, or when you create a new release.

To finalize the set-up for publishing to PyPI, see here. For activating the automatic documentation with MkDocs, see here. To enable the code coverage reports, see here.

Releasing a new version

• Create an API Token on PyPI.
• Add the API Token to your projects secrets with the name PYPI_TOKEN by visiting this page.
• Create a new release on Github.
• Create a new tag in the form *.*.*.

For more details, see here.

---

Repository initiated with fpgmaas/cookiecutter-uv.