gluekit 1.0.1.dev1

A developer‑centric Glue CLI built to speed local iteration with git/uv‑style commands, while bridging to highly controlled (eg Fedramped) AWS consoles for PII‑sensitive runs and smoothing onboarding around Glue’s quirks—like the lack of script‑to‑notebook conversion and the friction of refactoring Glue scripts.

gluekit

Introduction

gluekit is reusable AWS Glue job tooling for local-first development. It is designed for teams that want Glue job definitions, scripts, notebooks, and supporting artifacts in source control while still being able to pull from AWS, edit safely, validate locally, and push intentional changes back.

The command line is meant to feel closer to git and uv than to a cloud console: quick local commands, explicit artifact layouts, checkout-style context, and conversion workflows that smooth over AWS Glue quirks without hiding the underlying AWS behavior. A major motivation is making script and notebook work interchangeable enough for day-to-day development while still respecting that Glue scripts and Glue interactive notebooks are not identical runtime surfaces.

Local Development Emulators: gluekit provides scoped awsglue and pyspark emulators during local runs, using SQLFrame when available, so you can test jobs rapidly on your local machine without AWS compute. See the Local Development documentation for more details.

Suggested workflows

For AWS CLI-oriented setup, sync, and deployment recipes, start with the AWS CLI recipes folder, especially docs/aws_cli_recipes/workflows.md.

Common starting points:

• Pull a real Glue job into source control, edit the local script/config, and push selected artifacts back with gluekit pull, gluekit checkout, gluekit edit, and gluekit push.
• Use gluekit local setup, gluekit local s3, gluekit local ssm, and gluekit run to build a repeatable local test fixture before touching AWS.
• Keep account-specific values such as IAM roles, temp buckets, and script buckets in profile-scoped saved params so local configs can move between dev/prod-like AWS profiles without hand-editing generated artifacts.

Script-based development with notebook interoperability

gluekit supports a script-first workflow that can still interoperate with notebooks. Python scripts can use #%% cell tags to mark notebook-like cells. These tags divide a .py file into executable notebook cells; Markdown cells can use #%% [markdown]. This follows the same family of conventions described by VS Code's Python Interactive/Jupyter documentation, where Python files can contain Jupyter-style cell delimiters and be converted back into notebooks.

The intent is flexibility: you can develop in a normal Python file, use editor support for cell-by-cell execution, keep clean script diffs in git, and generate a Glue notebook when that is the deployment or debugging surface you need.

If you have already selected the job with checkout:

gluekit checkout {jobname}
gluekit convert --from script --to notebook

If you have not checked out the job, pass it explicitly:

gluekit convert {jobname} --from script --to notebook

Gluekit vs AWS CLI terminology

Some gluekit commands look like AWS CLI commands, but the scope is different:

• gluekit pull, gluekit push, gluekit build --push, and gluekit init use real AWS Glue/S3 APIs. On those commands, --profile means an AWS CLI credential profile, the same profile name boto3 would use.
• gluekit checkout --profile and gluekit config set --profile update local gluekit state in .gluekit/glue_set.json. The value is a gluekit profile scope for saved params; it usually matches an AWS profile name, but those commands do not contact AWS.
• gluekit local ... and gluekit run are local test tooling. Their S3, SSM, region, job, and profile values are labels or mocked fixture values stored in JSON and used by local moto/boto3 mocks. They do not create, read, or delete real AWS resources.

Install

If using uv as a global tool (recommended)

Ensure you have uv installed globally (see uv installation docs) and then run:

uv tool install https://github.com/NORCUChicago/gluekit.git

If using pip:

pip install https://github.com/NORCUChicago/gluekit.git

Quick start

1. Install the AWS command line tool and configure your credentials. If using sso, need to log in with

aws sso login --profile your-profile-name

2. Right now, the best way to get started is to check out the Command line help documentation below, which is auto-generated from the live CLI --help output and includes usage examples for each command.

Quick start commands

Glue job management

Initialize a workspace from your AWS CLI credential profiles:

gluekit init

Pull or check out the job you want to work on, then make focused config changes:

gluekit checkout my-job --profile survey-dev
gluekit edit --role arn:aws:iam::111111111111:role/glue-dev --temp-dir s3://dev-bucket/tmp/

Add local artifacts or shared files to the checked-out job:

gluekit add dist/my_pkg-0.0.1-py3-none-any.whl --package-whl
gluekit add glue/lib/helpers.py --as-path

Convert between Glue scripts and notebooks when your workflow changes:

gluekit convert my-job --from script --to notebook
gluekit convert my-job --from notebook --to script

Build Python artifacts and optionally push the checked-out job updates:

gluekit build
gluekit build --bump-version patch --package-whl --push

Local Glue job development

Create and check out the local setup with mocked AWS resource mappings:

gluekit local setup \
  --profile local-dev \
  --s3-root sources/s3 \
  --s3-bucket local-dev=tests/fixtures \
  --ssm-param /app/runtime=local

Inspect the saved setup, add SSM parameters, and run the Glue script locally:

gluekit local status
gluekit local ssm put /app/runtime local
gluekit run glue/scripts/my-job.py --glue-arg run_date=2026-06-18

Use --s3-root sources/s3 to map an entire mocked profile/account S3 namespace, where sources/s3/{bucket}/{prefix/key} appears as s3://{bucket}/{prefix/key}. Use --s3-bucket as {s3 bucket in mocked profile}={local directory} for a single bucket, and --ssm-param as {ssm parameter in mocked profile}={local parameter value}.

Local setup writes fixture data to .gluekit/local.json and records the active local settings under checkout.local in .gluekit/glue_set.json. The local --profile value is saved in both places as a cross-reference label for comparison and validation; gluekit local setup does not authenticate to AWS.

Generated metadata

gluekit now includes two checked-in metadata artifacts used by notebook conversion and Glue job validation helpers:

• src/gluekit/resources/glue_job_schema.json is generated from the local botocore AWS Glue service model.
• src/gluekit/resources/magic_map.json is a curated mapping of Glue job config fields to AWS Glue Interactive Sessions magics.

The schema artifact is generated by:

uv run python scripts/extract_glue_schema.py

Rerun that script when changing schema-driven validation logic, when upgrading boto3/botocore in a way that could change the Glue service model, or when you need to refresh the checked-in schema after AWS Glue API changes.

The magic map is intentionally maintained by hand because it reflects documented notebook behavior rather than raw AWS API structure. When updating notebook conversion behavior, review the AWS Glue Interactive Sessions magics docs and keep src/gluekit/resources/magic_map.json in sync.

edit workflow

gluekit edit is checkout-first. The default flow is to select a job once with gluekit checkout my-job and then run gluekit edit ... against that active local context.

Use gluekit add when you want the older inferred artifact behavior for local paths, wheels, zip files, and remote module specifiers. Use gluekit update when you need a raw property-path setter. Both remain available for compatibility, but gluekit edit is the primary mutation surface.

For the common package-wheel flow, run gluekit add --package-whl after building the project. It uses the single wheel matching dist/*.whl, routes it to --additional-python-modules, replaces the previously tracked dist/*.whl package wheel, updates SourceControlDetails.AdditionalPythonFiles, and refreshes an existing notebook's generated Glue magic cells.

If you need to target a different local config for one command, pass the job name directly to gluekit edit <job-name> ..., or keep using gluekit add --job-name other-job <items...> for the compatibility workflow. That stays parallel to GitHub CLI's context-first workflow, where a flag like --repo overrides the ambient target without changing the saved default context.

Profile-specific params and build automation

Use profile-scoped saved params for account-dependent values such as IAM roles, script buckets, temp buckets, and other Glue default arguments. These are local gluekit param scopes saved in .gluekit/glue_set.json; they are only used as real AWS credential profiles by commands that call AWS, such as pull, push, build --push, and init.

gluekit config set my-job --profile survey-dev --Role=arn:aws:iam::111111111111:role/glue-dev
gluekit config set my-job --profile survey-prod --Role=arn:aws:iam::222222222222:role/glue-prod
gluekit config set my-job --profile survey-prod --command_script_location=s3://prod-bucket/scripts/my-job.py

Then check out the job and gluekit profile scope together:

gluekit checkout my-job --profile survey-prod

After a job is checked out, switch only the active profile scope with:

gluekit checkout --profile survey-dev

Checkout stores the active gluekit profile scope in .gluekit/glue_set.json and applies the matching saved params to the local config immediately, including mapped Glue config paths such as Role, Command.ScriptLocation, DefaultArguments.*, and SourceControlDetails.*. gluekit pull validates existing active profile mappings against the pulled config; if the active profile has no mappings yet but another profile exists for the job, pull detects and saves role plus S3/config mappings using the same profile comparison rules as gluekit init. gluekit push validates that the active profile mappings already align with the local config before uploading, and reports every mismatch instead of rewriting the config at push time. gluekit pull, gluekit push, and gluekit build --push also use the active profile value as the AWS CLI credential profile when --profile is not passed.

The common package release loop can now be a single command:

gluekit build --bump-version patch --package-whl --push

That bumps pyproject.toml, builds artifacts, refreshes the checked-out job's tracked dist/*.whl in --additional-python-modules, applies active profile params before deriving S3 paths, and pushes the selected Glue artifacts/config. Use --dry-run to preview the build, wheel update, and push steps.

For first-time repository setup, gluekit init scans your AWS CLI config profiles, prompts for the Glue job to inspect in each selected profile, can save profile-specific role and S3 params locally, and sets the first checked-out job/profile scope.

Command line help documentation

Auto-generated from live CLI --help output.

gluekit --help

Usage: gluekit [OPTIONS] COMMAND [ARGS]...

Options:
  --install-completion  Install completion for the current shell.
  --show-completion     Show completion for the current shell, to copy it or
                        customize the installation.
  --help                Show this message and exit.

Commands:
  build     Build local artifacts; only touches AWS when --push is used.
  checkout  Replace the active local checkout selection.
  edit      Edit one local Glue job config with explicit mutation flags.
  clone     Clone local Glue job configurations and artifacts.
  set       Set reusable local key/value parameters for a Glue job.
  convert   Convert Glue jobs between script and notebook formats.
  init      Initialize local repo-level Gluekit settings from AWS CLI...
  list      List real AWS Glue jobs in the configured AWS account.
  pull      Download real AWS Glue job artifacts and config into the local...
  push      Upload local artifacts to real AWS Glue/S3 and optionally...
  run       Run a Glue script locally with emulated Glue libraries and...
  remove    Remove local Glue job configs and local artifacts.
  status    Show the active local checkout and its local-to-S3 artifact...
  sync      Sync generated Glue config metadata cells in local...
  add       Compatibility command for inferred artifact and module additions.
  update    Compatibility command for generic property updates.
  config    Show and set local reusable Glue config parameters.
  local     Manage local-only Glue development setups and mocked AWS fixtures.

gluekit pull --help

Usage: gluekit pull [OPTIONS] [JOB_NAME] [TARGET_JOB]

  Download real AWS Glue job artifacts and config into the local workspace.

Arguments:
  [JOB_NAME]    Glue job name or pattern to pull. Use "*" for all jobs.
  [TARGET_JOB]  Optional local job name to write pulled artifacts to.

Options:
  -f, --force                     Download even if the local config is newer or
                                  equal to LastModifiedOn.
  --dry-run                       Show what would be downloaded without writing
                                  files.
  -i, --include TEXT              Include only specific components (config,
                                  script, notebook, extra-py-files, extra-
                                  files).
  -x, --exclude TEXT              Exclude specific components (config, script,
                                  notebook, extra-py-files, extra-files).
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  --write-job-list / --no-write-job-list
                                  Write glue/glue_full_job_list.csv during the
                                  run.  [default: write-job-list]
  -p, --profile TEXT              AWS CLI credential profile used for real AWS
                                  Glue/S3 API calls; not a gluekit local test
                                  profile.
  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
                                  sso login' when credentials are missing or
                                  expired.  [default: auto-login]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit pull my-job --profile my-sso-profile`

  - `gluekit pull source-job target-job --profile my-sso-profile`

  - `gluekit pull "my-job-\*" --force --include config,script --exclude
  notebook`

  - `gluekit pull "\*" --dry-run --profile my-sso-profile --no-auto-login`

gluekit edit --help

Usage: gluekit edit [OPTIONS] [JOB_NAME]

  Edit one local Glue job config with explicit mutation flags.

Arguments:
  [JOB_NAME]  Glue job name to edit. Defaults to the active checkout selection.

Options:
  --description TEXT              Set the job description.
  --role TEXT                     Set the IAM role for the job.
  --glue-version TEXT             Set the Glue version.
  --worker-type TEXT              Set the worker type.
  --number-of-workers INTEGER     Set the number of workers.
  --timeout INTEGER               Set the job timeout in minutes.
  --max-retries INTEGER           Set the max retry count.
  --execution-class TEXT          Set the execution class.
  --max-concurrent-runs INTEGER   Set the max concurrent runs.
  --job-run-queuing-enabled / --no-job-run-queuing-enabled
                                  Enable or disable job run queuing.
  --security-configuration TEXT   Set the security configuration name.
  --maintenance-window TEXT       Set the maintenance window.
  --log-uri TEXT                  Set the log URI.
  --command-name TEXT             Set Command.Name.
  --python-version TEXT           Set Command.PythonVersion.
  --runtime TEXT                  Set Command.Runtime.
  --remote-script-location TEXT   Set Command.ScriptLocation.
  --local-script-location TEXT    Set SourceControlDetails.ScriptLocation.
  --notebook-location TEXT        Set SourceControlDetails.NotebookLocation and
                                  NotebookPath.
  --source-control-provider TEXT  Set SourceControlDetails.Provider.
  --source-control-owner TEXT     Set SourceControlDetails.Owner.
  --source-control-repository TEXT
                                  Set SourceControlDetails.Repository.
  --source-control-branch TEXT    Set SourceControlDetails.Branch.
  --source-control-folder TEXT    Set SourceControlDetails.Folder.
  --add-connection TEXT           Add one or more Glue connections.
  --remove-connection TEXT        Remove one or more Glue connections.
  --add-additional-python-modules TEXT
                                  Add one or more values to --additional-python-
                                  modules.
  --remove-additional-python-modules TEXT
                                  Remove one or more values from --additional-
                                  python-modules.
  --add-extra-py-files TEXT       Track one or more local relative Python files
                                  in SourceControlDetails.ExtraPyFiles and
                                  --extra-py-files.
  --remove-extra-py-files TEXT    Remove one or more local relative Python files
                                  from SourceControlDetails.ExtraPyFiles and
                                  --extra-py-files.
  --add-extra-jars TEXT           Add one or more values to --extra-jars.
  --remove-extra-jars TEXT        Remove one or more values from --extra-jars.
  --add-additional-python-files TEXT
                                  Track one or more local paths in
                                  SourceControlDetails.AdditionalPythonFiles and
                                  --extra-py-files.
  --remove-additional-python-files TEXT
                                  Remove one or more local paths from
                                  SourceControlDetails.AdditionalPythonFiles and
                                  --extra-py-files.
  --add-extra-files TEXT          Track one or more local paths in
                                  SourceControlDetails.ExtraFiles and --extra-
                                  files.
  --remove-extra-files TEXT       Remove one or more local paths from
                                  SourceControlDetails.ExtraFiles and --extra-
                                  files.
  --add-default-argument TEXT     Set a DefaultArguments entry using KEY=VALUE.
                                  Repeat as needed.
  --remove-default-argument TEXT  Remove a DefaultArguments entry by key. Repeat
                                  as needed.
  --add-non-overridable-argument TEXT
                                  Set a NonOverridableArguments entry using
                                  KEY=VALUE. Repeat as needed.
  --remove-non-overridable-argument TEXT
                                  Remove a NonOverridableArguments entry by key.
                                  Repeat as needed.
  --enable-metrics / --disable-metrics
                                  Set DefaultArguments.--enable-metrics to true
                                  or false.
  --enable-observability-metrics / --disable-observability-metrics
                                  Set DefaultArguments.--enable-observability-
                                  metrics to true or false.
  --enable-glue-datacatalog / --disable-glue-datacatalog
                                  Set DefaultArguments.--enable-glue-datacatalog
                                  to true or false.
  --enable-continuous-cloudwatch-log / --disable-continuous-cloudwatch-log
                                  Set DefaultArguments.--enable-continuous-
                                  cloudwatch-log to true or false.
  --enable-spark-ui / --disable-spark-ui
                                  Set DefaultArguments.--enable-spark-ui to true
                                  or false.
  --enable-job-insights / --disable-job-insights
                                  Set DefaultArguments.--enable-job-insights to
                                  true or false.
  --enable-auto-scaling / --disable-auto-scaling
                                  Set DefaultArguments.--enable-auto-scaling to
                                  true or false.
  --job-bookmark-option TEXT      Set DefaultArguments.--job-bookmark-option.
  --temp-dir TEXT                 Set DefaultArguments.--TempDir.
  --spark-event-logs-path TEXT    Set DefaultArguments.--spark-event-logs-path.
  --datalake-formats TEXT         Set DefaultArguments.--datalake-formats.
  --spark-conf TEXT               Set DefaultArguments.--conf.
  --continuous-log-log-group TEXT
                                  Set DefaultArguments.--continuous-log-
                                  logGroup.
  --continuous-log-log-stream-prefix TEXT
                                  Set DefaultArguments.--continuous-log-
                                  logStreamPrefix.
  --continuous-log-conversion-pattern TEXT
                                  Set DefaultArguments.--continuous-log-
                                  conversionPattern.
  --executor-cores INTEGER        Set DefaultArguments.--executor-cores.
  --customer-driver-env-vars TEXT
                                  Set DefaultArguments.--customer-driver-env-
                                  vars.
  --customer-executor-env-vars TEXT
                                  Set DefaultArguments.--customer-executor-env-
                                  vars.
  --python-modules-installer-option TEXT
                                  Set DefaultArguments.--python-modules-
                                  installer-option.
  --dry-run                       Show what would be updated without writing
                                  files.
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit edit my-job --description 'Updated job description' --glue-version
  5.0`

  - `gluekit edit --add-additional-python-files glue/libs/helper.py --add-extra-
  files glue/resources/job.json`

  - `gluekit edit my-job --temp-dir s3://bucket/tmp --remove-default-argument
  legacy-arg --enable-metrics`

  - `gluekit edit my-job --enable-auto-scaling --enable-job-insights --executor-
  cores 8`

gluekit add --help

Usage: gluekit add [OPTIONS] [ARGS]...

  Compatibility command for inferred artifact and module additions. Prefer edit
  for explicit config changes.

Arguments:
  [ARGS]...  Paths or module specifiers to add to the selected Glue job config.

Options:
  --job-name TEXT    Override the active checked-out job for this command only.
  --as-path          Treat all items as local paths.
  --as-pypi          Treat all items as PyPI modules.
  --package-whl      Use the package wheel from dist/*.whl.
  --dry-run          Show what would be updated without writing files.
  --config-dir PATH  Directory containing Glue job config files.  [default:
                     glue/configs]
  --help             Show this message and exit.

  **Examples**

  - `gluekit checkout my-job`

  - `gluekit add dist/my-package.whl`

  - `gluekit add --job-name other-job requests==2.32.3 --as-pypi`

gluekit convert --help

Usage: gluekit convert [OPTIONS] [JOB_NAME]

  Convert Glue jobs between script and notebook formats.

Arguments:
  [JOB_NAME]  Glue job name to convert. Defaults to the active checkout
              selection.

Options:
  --from TEXT                   Source format (script or notebook).  [default:
                                script]
  --to TEXT                     Target format (script or notebook).  [default:
                                notebook]
  --use-config / --script-only  Use config metadata (magics and paths) when
                                available.  [default: use-config]
  --dry-run                     Show what would be converted without writing
                                files.
  --config-dir PATH             Directory containing Glue job config files.
                                [default: glue/configs]
  --help                        Show this message and exit.

  **Examples**

  - `gluekit convert my-job --from script --to notebook --use-config`

  - `gluekit convert my-job --from notebook --to script --script-only`

gluekit sync --help

Usage: gluekit sync [OPTIONS] [JOB_NAME]

  Sync generated Glue config metadata cells in local script/notebook files.

Arguments:
  [JOB_NAME]  Glue job name to sync. Defaults to the active checkout selection.

Options:
  --dry-run          Show what would be updated without writing files.
  --config-dir PATH  Directory containing Glue job config files.  [default:
                     glue/configs]
  --help             Show this message and exit.

  **Examples**

  - `gluekit sync my-job`

  - `gluekit sync my-job --dry-run`

gluekit status --help

Usage: gluekit status [OPTIONS]

  Show the active local checkout and its local-to-S3 artifact mappings.

Options:
  --config-dir PATH  Directory containing Glue job config files.  [default:
                     glue/configs]
  --help             Show this message and exit.

  **Examples**

  - `gluekit status`

  - `gluekit status --config-dir glue/configs`

gluekit run --help

Usage: gluekit run [OPTIONS] SCRIPT_PATH

  Run a Glue script locally with emulated Glue libraries and mocked AWS
  services.

Arguments:
  SCRIPT_PATH  Path to the Glue Python script to execute locally.  [required]

Options:
  --glue-arg TEXT       Glue script argument in KEY=VALUE form; passed through
                        as --KEY VALUE.
  --job-name TEXT       Default JOB_NAME used when not provided in script args.
                        [default: local-glue-job]
  --create-bucket TEXT  Create a mocked S3 bucket only inside the local run.
                        Repeat as needed.
  --ssm-parameter TEXT  Seed a mocked SSM parameter only inside the local run as
                        NAME=VALUE. Repeat as needed.
  --aws-region TEXT     Mock AWS region used for local boto3 clients. Defaults
                        to us-east-1.
  --config-file FILE    Local fixture config file for mocked S3/SSM. Defaults to
                        .gluekit/local.json when local settings are checked out.
  --help                Show this message and exit.

  **Examples**

  - `gluekit run ../ppsc-survey/glue/scripts/job_a.py --create-bucket my-input-
  bucket`

  - `gluekit run glue/scripts/job-a.py --ssm-parameter /app/env=dev --glue-arg
  report_date=2026-06-18`

  - `gluekit run glue/scripts/job-a.py -- --JOB_NAME custom-local-job
  --report_date 2026-06-18`

gluekit local --help

Usage: gluekit local [OPTIONS] COMMAND [ARGS]...

  Manage local-only Glue development setups and mocked AWS fixtures.

Options:
  --help  Show this message and exit.

Commands:
  setup   Create or update and check out the local-only development setup.
  status  Show the active local development setup.
  s3      Manage local mocked S3 fixture mappings; does not call AWS S3.
  ssm     Manage local mocked SSM Parameter Store values; does not call AWS...

gluekit local setup --help

Usage: gluekit local setup [OPTIONS]

  Create or update and check out the local-only development setup.

Options:
  --profile TEXT    Optional gluekit/AWS profile name to associate with this
                    local setup in checkout state and .gluekit/local.json; this
                    command does not authenticate to AWS.
  --s3-bucket TEXT  Map a mocked S3 bucket to a local directory in {s3 bucket in
                    mocked profile}={local directory} form. Repeat as needed.
  --s3-root PATH    Map a local directory as the full mocked S3 namespace for
                    this setup/profile. Expected layout: {local s3
                    root}/{bucket}/{prefix/key}.
  --ssm-param TEXT  Map a mocked SSM parameter to a local value in {ssm
                    parameter in mocked profile}={local parameter value} form.
                    Repeat as needed.
  --help            Show this message and exit.

  **Examples**

  - `gluekit local setup --profile local-dev --s3-root sources/s3`

  - `gluekit local setup --s3-bucket local-dev=tests/fixtures`

  - `gluekit local setup --ssm-param /app/runtime=local`

gluekit local status --help

Usage: gluekit local status [OPTIONS]

  Show the active local development setup.

Options:
  --help  Show this message and exit.

  **Examples**

  - `gluekit local status`

gluekit local s3 --help

Usage: gluekit local s3 [OPTIONS] COMMAND [ARGS]...

  Manage local mocked S3 fixture mappings; does not call AWS S3.

Options:
  --help  Show this message and exit.

Commands:
  cp       Copy a local file or mocked S3 object in the fixture config;...
  mount    Map a local directory recursively into mocked S3 keys; does not...
  unmount  Remove a recursive local-directory-to-mocked-S3 mapping.
  mv       Move a local file or mocked S3 object in the fixture config;...
  rm       Remove mapped mocked S3 objects from the local fixture config.
  ls       List mocked S3 buckets inferred from mappings, or mapped objects.

gluekit local s3 cp --help

Usage: gluekit local s3 cp [OPTIONS] SOURCE DESTINATION

  Copy a local file or mocked S3 object in the fixture config; does not call
  AWS.

Arguments:
  SOURCE       Local path or mocked S3 URI source.  [required]
  DESTINATION  Local path or mocked S3 URI destination.  [required]

Options:
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 cp tests/fixtures/input.json s3://my-bucket/input.json`

  - `gluekit local s3 cp s3://my-bucket/input.json s3://my-bucket/copy.json`

  - `gluekit local s3 cp s3://my-bucket/input.json tmp/input.json`

gluekit local s3 mount --help

Usage: gluekit local s3 mount [OPTIONS] SOURCE_DIR DESTINATION

  Map a local directory recursively into mocked S3 keys; does not call AWS.

Arguments:
  SOURCE_DIR   Local directory to map recursively.  [required]
  DESTINATION  Mocked S3 bucket or prefix base for mapped files.  [required]

Options:
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 mount tests/fixtures s3://local-dev/input/`

  - `gluekit local s3 mount data/mock s3://bucket-a`

gluekit local s3 unmount --help

Usage: gluekit local s3 unmount [OPTIONS] TARGET

  Remove a recursive local-directory-to-mocked-S3 mapping.

Arguments:
  TARGET  Local source directory or mocked S3 destination to remove.  [required]

Options:
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 unmount tests/fixtures`

  - `gluekit local s3 unmount s3://local-dev/input/`

gluekit local s3 mv --help

Usage: gluekit local s3 mv [OPTIONS] SOURCE DESTINATION

  Move a local file or mocked S3 object in the fixture config; does not call
  AWS.

Arguments:
  SOURCE       Local path or mocked S3 URI source.  [required]
  DESTINATION  Local path or mocked S3 URI destination.  [required]

Options:
  --delete-local      Delete the local source file after mapping a LocalPath to
                      S3.
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 mv s3://my-bucket/input.json s3://my-bucket/renamed.json`

  - `gluekit local s3 mv s3://my-bucket/input.json tmp/input.json`

gluekit local s3 rm --help

Usage: gluekit local s3 rm [OPTIONS] URI

  Remove mapped mocked S3 objects from the local fixture config.

Arguments:
  URI  Mapped mocked S3 object or prefix to remove.  [required]

Options:
  -r, --recursive     Remove all mapped objects under the S3 prefix.
  --config-file PATH  Local fixture config file to update. Defaults to
                      .gluekit/local.json.
  --help              Show this message and exit.

  **Examples**

  - `gluekit local s3 rm s3://my-bucket/input.json`

  - `gluekit local s3 rm s3://my-bucket/prefix/ --recursive`

gluekit local ssm --help

Usage: gluekit local ssm [OPTIONS] COMMAND [ARGS]...

  Manage local mocked SSM Parameter Store values; does not call AWS SSM.

Options:
  --help  Show this message and exit.

Commands:
  put  Create or update a mocked SSM parameter with local-only syntax.
  get  Show one mocked SSM parameter with local-only syntax.
  rm   Delete one mocked SSM parameter with local-only syntax.
  ls   List mocked SSM parameters with local-only syntax.

gluekit local ssm put --help

Usage: gluekit local ssm put [OPTIONS] NAME VALUE

  Create or update a mocked SSM parameter with local-only syntax.

Arguments:
  NAME   Mocked SSM parameter name.  [required]
  VALUE  Mocked SSM parameter value.  [required]

Options:
  --overwrite / --no-overwrite  Overwrite an existing parameter.  [default:
                                overwrite]
  --config-file PATH            Local fixture config file to update. Defaults to
                                .gluekit/local.json.
  --help                        Show this message and exit.

  **Examples**

  - `gluekit local ssm put /app/env dev`

gluekit build --help

Usage: gluekit build [OPTIONS]

  Build local artifacts; only touches AWS when --push is used.

Options:
  --job-name TEXT                 Checked-out or explicit local Glue job to
                                  update when using --package-whl or --push.
  --dry-run                       Show what would be built and zipped without
                                  running build commands.
  --bump-version, --bump TEXT     Bump pyproject.toml [project].version before
                                  building: major, minor, patch, or X.Y.Z.
  --build-tool TEXT               Build frontend to use: auto, uv, or build.
                                  [default: auto]
  --out-dir PATH                  The output directory to which distributions
                                  should be written.
  --no-version-suffix             Remove the version suffix from generated zip
                                  artifact names.
  -v, --verbose                   Print build commands before execution.
  --package-whl                   After building, replace the tracked dist/*.whl
                                  in the selected Glue config.
  --push                          Push the selected Glue config and artifacts
                                  after building.
  -i, --include TEXT              For --push, include only specific components
                                  (script, notebook, additional-python-modules,
                                  extra-files, job-config).
  -x, --exclude TEXT              For --push, exclude specific components
                                  (script, notebook, additional-python-modules,
                                  extra-files, job-config).
  --update-config / --no-update-config
                                  For --push, update Glue job configuration
                                  after uploading files.  [default: update-
                                  config]
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  -p, --profile TEXT              AWS CLI credential profile for profile-scoped
                                  config params and, with --push, real AWS
                                  Glue/S3 API calls.
  --auto-login / --no-auto-login  For --push with a real AWS profile,
                                  automatically run 'aws sso login' when
                                  credentials are missing or expired.  [default:
                                  auto-login]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit build`

  - `gluekit build --build-tool uv`

  - `gluekit build --out-dir build/artifacts --no-version-suffix`

  - `gluekit build --build-tool build --dry-run`

gluekit push --help

Usage: gluekit push [OPTIONS] [JOB_NAME] [TARGET_JOB]

  Upload local artifacts to real AWS Glue/S3 and optionally update the job
  config.

Arguments:
  [JOB_NAME]    Glue job name or pattern to push. Use "*" for all configs.
  [TARGET_JOB]  Optional remote Glue job name to push to.

Options:
  --dry-run                       Show what would be uploaded without writing
                                  files.
  -i, --include TEXT              Include only specific components (script,
                                  notebook, additional-python-modules, extra-
                                  files, job-config).
  -x, --exclude TEXT              Exclude specific components (script, notebook,
                                  additional-python-modules, extra-files, job-
                                  config).
  --update-config / --no-update-config
                                  Update Glue job configuration after uploading
                                  files.  [default: update-config]
  --build                         Build local package artifacts in the workspace
                                  root before push.
  --build-tool TEXT               Build frontend for --build: auto, uv, or
                                  build.  [default: auto]
  -v, --verbose                   Print local build command(s) before execution.
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  -p, --profile TEXT              AWS CLI credential profile used for real AWS
                                  Glue/S3 API calls; not a gluekit local test
                                  profile.
  --auto-login / --no-auto-login  For real AWS profiles, automatically run 'aws
                                  sso login' when credentials are missing or
                                  expired.  [default: auto-login]
  --help                          Show this message and exit.

  **Examples**

  - `gluekit push my-job --profile my-sso-profile`

  - `gluekit push source-job target-job --profile my-sso-profile`

  - `gluekit push "my-job-\*" --include script,job-config --exclude notebook`

  - `gluekit push my-job --build --build-tool auto --dry-run`

gluekit set --help

Usage: gluekit set [OPTIONS] [JOB_NAME]

  Set reusable local key/value parameters for a Glue job.

Arguments:
  [JOB_NAME]  Glue job name to save params for.

Options:
  --global            Save params as global defaults for all jobs.
  -p, --profile TEXT  Save params under this gluekit profile scope, usually
                      matching an AWS CLI profile name; does not contact AWS.
  --help              Show this message and exit.

  **Examples**

  - `gluekit set my-job --script_location glue/scripts/my-job.py --extra_files
  s3://my-bucket/shared/config.json`

  - `gluekit pull my-job --profile my-sso-profile`

  - `gluekit set --global --additional_python_modules "pydantic==2.11.7"`

  - `gluekit push "my-job-\*" --profile my-sso-profile`

  - `gluekit config set my-job --default_arguments.--TempDir s3://my-
  bucket/tmp/`

  - `gluekit config get my-job`

  - `gluekit config get`

  - `gluekit set my-job --extra_py_files s3://my-
  bucket/dist/gluekit-0.0.1-py3-none-any.whl`

  - `gluekit push my-job --build --build-tool auto --profile my-sso-profile`

gluekit config --help

Usage: gluekit config [OPTIONS] COMMAND [ARGS]...

  Show and set local reusable Glue config parameters.

Options:
  --help  Show this message and exit.

Commands:
  set  Set reusable local Glue config params by job or globally.
  get  Show stored local Glue config params.

gluekit config set --help

Usage: gluekit config set [OPTIONS] [JOB_NAME]

  Set reusable local Glue config params by job or globally.

Arguments:
  [JOB_NAME]  Glue job name to save params for.

Options:
  --global            Save params as global defaults for all jobs.
  -p, --profile TEXT  Save params under this gluekit profile scope, usually
                      matching an AWS CLI profile name; does not contact AWS.
  --help              Show this message and exit.

  **Examples**

  - `gluekit config set my-job --default_arguments.--TempDir s3://my-
  bucket/tmp/`

  - `gluekit config set --global --additional_python_modules "pydantic==2.11.7"`

gluekit config get --help

Usage: gluekit config get [OPTIONS] [JOB_NAME]

  Show stored local Glue config params.

Arguments:
  [JOB_NAME]  Optional Glue job name to view merged params (global + job).

Options:
  -p, --profile TEXT  Include params from this gluekit profile scope, usually
                      matching an AWS CLI profile name; does not contact AWS.
  --help              Show this message and exit.

  **Examples**

  - `gluekit config get my-job`

  - `gluekit config get`

gluekit list --help

Usage: gluekit list [OPTIONS]

  List real AWS Glue jobs in the configured AWS account.

Options:
  --help  Show this message and exit.

  **Examples**

  - `gluekit list --help`

  - `gluekit init --help`

gluekit init --help

Usage: gluekit init [OPTIONS]

  Initialize local repo-level Gluekit settings from AWS CLI profiles.

Options:
  --aws-config PATH               Local AWS CLI config file to scan for
                                  credential profile names; defaults to
                                  AWS_CONFIG_FILE or ~/.aws/config.
  --config-dir PATH               Directory containing Glue job config files.
                                  [default: glue/configs]
  --auto-login / --no-auto-login  For selected real AWS profiles, automatically
                                  run 'aws sso login' when credentials are
                                  missing or expired.  [default: auto-login]
  --detect-roles / --no-detect-roles
                                  Fetch the selected real AWS Glue job in each
                                  profile and save its Role as local gluekit
                                  params.  [default: detect-roles]
  --detect-buckets / --no-detect-buckets
                                  Compare S3 URIs in real AWS Glue job configs
                                  across profiles and save local gluekit profile
                                  mappings.  [default: detect-buckets]
  --match-threshold INTEGER       Minimum rapidfuzz score for non-exact S3 key
                                  matches.  [default: 85]
  --baseline-profile TEXT         Selected AWS CLI profile to store as the first
                                  gluekit checkout scope and S3 comparison
                                  baseline.
  --profile-job TEXT              Real AWS Glue job to inspect for an AWS CLI
                                  profile, as PROFILE=GLUE_JOB_NAME. May be
                                  repeated.
  --help                          Show this message and exit.

  **Examples**

  - `gluekit init`

Getting started with your project

1. Create a New Repository

First, create a repository on GitHub with the same name as this project, and then run the following commands:

git init -b main
git add .
git commit -m "init commit"
git remote add origin [e-mail]:mbkranz/gluekit.git
git push -u origin main

2. Set Up Your Development Environment

Then, install the environment and the pre-commit hooks with

make install

This will also generate your uv.lock file

3. Run the pre-commit hooks

Initially, the CI/CD pipeline might be failing due to formatting issues. To resolve those run:

uv run pre-commit run -a

4. Commit the changes

Lastly, commit the changes made by the two steps above to your repository.

git add .
git commit -m 'Fix formatting issues'
git push origin main

You are now ready to start development on your project! The CI/CD pipeline will be triggered when you open a pull request, merge to main, or when you create a new release.

To finalize the set-up for publishing to PyPI, see here. For activating the automatic documentation with MkDocs, see here. To enable the code coverage reports, see here.

Releasing a new version

• Create an API Token on PyPI.
• Add the API Token to your projects secrets with the name PYPI_TOKEN by visiting this page.
• Create a new release on Github.
• Create a new tag in the form *.*.*.

For more details, see here.

---

Repository initiated with fpgmaas/cookiecutter-uv.