Skip to main content

A command line tool to help deploy AWS Glue Jobs at ease :)

Project description

glueetl

A command line tool to help deploy AWS Glue Jobs at ease :)

Install

$ pip install glueetl

How to develop a Glue job

You can develop a Glue job by following the steps below.

1. Set up AWS Credentials and Region

Before you can deploy a Glue job to AWS Glue, you must set up AWS Credentials and Region.

$ vim ~/.aws/credentials

[default]
aws_access_key_id=<AWS_ACCESS_KEY_ID>
aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>
region=<REGION>

2. Initialize a Glue job

$ mkdir sample
$ cd sample
$ glueetl init
.
├── README.md
├── config.yaml
└── script.py

config.yaml includes job properties and currently it supports the following properties:

job: 
  name: sample-glue-job
  role_name: AWSGlueServiceRole
  script_location: s3://glue-job-scripts/sample-glue-job/script.py
  max_concurrent_runs: 10
  command_name: glueetl
  max_retries: 0
  timeout: 28800
  max_capacity: 10
  connections:
    - first_connection
    - second_connection
  default_arguments:
    argument1: value1
    argument2: value2
  non_overridable_arguments:
    argument1: value1
    argument2: value2
  trigger:
    name: trigger-sample-glue-job
    schedule: cron(5 * * * ? *)
  tags:
    key1: value1
    key2: value2

Please change default values in file config.yaml and write your job logic in file script.py.

3. Deploy a Glue job

$ cd sample
$ glueetl deploy

Your job will be deployed to AWS Glue.

4. Run a Glue job

You can manually run your Glue job like this.

$ cd sample
$ glueetl run --arg1=value1 --arg2=value2

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for glueetl, version 0.0.5
Filename, size File type Python version Upload date Hashes
Filename, size glueetl-0.0.5.tar.gz (4.1 kB) File type Source Python version None Upload date Hashes View
Filename, size glueetl-0.0.5-py3-none-any.whl (5.5 kB) File type Wheel Python version py3 Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page