Skip to main content

The CDK Construct Library for AWS::Glue

Project description

AWS Glue Construct Library

---

cfn-resources: Stable

All classes with the Cfn prefix in this module (CFN Resources) are always stable and safe to use.

cdk-constructs: Experimental

The APIs of higher level constructs in this module are experimental and under active development. They are subject to non-backward compatible changes or removal in any future version. These are not subject to the Semantic Versioning model and breaking changes will be announced in the release notes. This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.


This module is part of the AWS Cloud Development Kit project.

Connection

A Connection allows Glue jobs, crawlers and development endpoints to access certain types of data stores. For example, to create a network connection to connect to a data source within a VPC:

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Connection(stack, "MyConnection",
    connection_type=glue.ConnectionTypes.NETWORK,
    # The security groups granting AWS Glue inbound access to the data source within the VPC
    security_groups=[security_group],
    # The VPC subnet which contains the data source
    subnet=subnet
)

If you need to use a connection type that doesn't exist as a static member on ConnectionType, you can instantiate a ConnectionType object, e.g: new glue.ConnectionType('NEW_TYPE').

See Adding a Connection to Your Data Store and Connection Structure documentation for more information on the supported data stores and their configurations.

Database

A Database is a logical grouping of Tables in the Glue Catalog.

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Database(stack, "MyDatabase",
    database_name="my_database"
)

SecurityConfiguration

A SecurityConfiguration is a set of security properties that can be used by AWS Glue to encrypt data at rest.

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.SecurityConfiguration(stack, "MySecurityConfiguration",
    security_configuration_name="name",
    cloud_watch_encryption={
        "mode": glue.CloudWatchEncryptionMode.KMS
    },
    job_bookmarks_encryption={
        "mode": glue.JobBookmarksEncryptionMode.CLIENT_SIDE_KMS
    },
    s3_encryption={
        "mode": glue.S3EncryptionMode.KMS
    }
)

By default, a shared KMS key is created for use with the encryption configurations that require one. You can also supply your own key for each encryption config, for example, for CloudWatch encryption:

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.SecurityConfiguration(stack, "MySecurityConfiguration",
    security_configuration_name="name",
    cloud_watch_encryption={
        "mode": glue.CloudWatchEncryptionMode.KMS,
        "kms_key": key
    }
)

See documentation for more info for Glue encrypting data written by Crawlers, Jobs, and Development Endpoints.

Table

A Glue table describes a table of data in S3: its structure (column names and types), location of data (S3 objects with a common prefix in a S3 bucket), and format for the files (Json, Avro, Parquet, etc.):

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Table(stack, "MyTable",
    database=my_database,
    table_name="my_table",
    columns=[{
        "name": "col1",
        "type": glue.Schema.STRING
    }, {
        "name": "col2",
        "type": glue.Schema.array(Schema.STRING),
        "comment": "col2 is an array of strings"
    }],
    data_format=glue.DataFormat.JSON
)

By default, a S3 bucket will be created to store the table's data but you can manually pass the bucket and s3Prefix:

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Table(stack, "MyTable",
    bucket=my_bucket,
    s3_prefix="my-table/", ...
)

By default, an S3 bucket will be created to store the table's data and stored in the bucket root. You can also manually pass the bucket and s3Prefix:

Partitions

To improve query performance, a table can specify partitionKeys on which data is stored and queried separately. For example, you might partition a table by year and month to optimize queries based on a time window:

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Table(stack, "MyTable",
    database=my_database,
    table_name="my_table",
    columns=[{
        "name": "col1",
        "type": glue.Schema.STRING
    }],
    partition_keys=[{
        "name": "year",
        "type": glue.Schema.SMALL_INT
    }, {
        "name": "month",
        "type": glue.Schema.SMALL_INT
    }],
    data_format=glue.DataFormat.JSON
)

Encryption

You can enable encryption on a Table's data:

  • Unencrypted - files are not encrypted. The default encryption setting.
  • S3Managed - Server side encryption (SSE-S3) with an Amazon S3-managed key.
# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Table(stack, "MyTable",
    encryption=glue.TableEncryption.S3_MANAGED, ...
)
  • Kms - Server-side encryption (SSE-KMS) with an AWS KMS Key managed by the account owner.
# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
# KMS key is created automatically
glue.Table(stack, "MyTable",
    encryption=glue.TableEncryption.KMS, ...
)

# with an explicit KMS key
glue.Table(stack, "MyTable",
    encryption=glue.TableEncryption.KMS,
    encryption_key=kms.Key(stack, "MyKey"), ...
)
  • KmsManaged - Server-side encryption (SSE-KMS), like Kms, except with an AWS KMS Key managed by the AWS Key Management Service.
# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Table(stack, "MyTable",
    encryption=glue.TableEncryption.KMS_MANAGED, ...
)
  • ClientSideKms - Client-side encryption (CSE-KMS) with an AWS KMS Key managed by the account owner.
# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
# KMS key is created automatically
glue.Table(stack, "MyTable",
    encryption=glue.TableEncryption.CLIENT_SIDE_KMS, ...
)

# with an explicit KMS key
glue.Table(stack, "MyTable",
    encryption=glue.TableEncryption.CLIENT_SIDE_KMS,
    encryption_key=kms.Key(stack, "MyKey"), ...
)

Note: you cannot provide a Bucket when creating the Table if you wish to use server-side encryption (KMS, KMS_MANAGED or S3_MANAGED).

Types

A table's schema is a collection of columns, each of which have a name and a type. Types are recursive structures, consisting of primitive and complex types:

# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
glue.Table(stack, "MyTable",
    columns=[{
        "name": "primitive_column",
        "type": glue.Schema.STRING
    }, {
        "name": "array_column",
        "type": glue.Schema.array(glue.Schema.INTEGER),
        "comment": "array<integer>"
    }, {
        "name": "map_column",
        "type": glue.Schema.map(glue.Schema.STRING, glue.Schema.TIMESTAMP),
        "comment": "map<string,string>"
    }, {
        "name": "struct_column",
        "type": glue.Schema.struct([
            name="nested_column",
            type=glue.Schema.DATE,
            comment="nested comment"
        ]),
        "comment": "struct<nested_column:date COMMENT 'nested comment'>"
    }], ...

Primitives

Numeric

Name Type Comments
FLOAT Constant A 32-bit single-precision floating point number
INTEGER Constant A 32-bit signed value in two's complement format, with a minimum value of -2^31 and a maximum value of 2^31-1
DOUBLE Constant A 64-bit double-precision floating point number
BIG_INT Constant A 64-bit signed INTEGER in two’s complement format, with a minimum value of -2^63 and a maximum value of 2^63 -1
SMALL_INT Constant A 16-bit signed INTEGER in two’s complement format, with a minimum value of -2^15 and a maximum value of 2^15-1
TINY_INT Constant A 8-bit signed INTEGER in two’s complement format, with a minimum value of -2^7 and a maximum value of 2^7-1

Date and time

Name Type Comments
DATE Constant A date in UNIX format, such as YYYY-MM-DD.
TIMESTAMP Constant Date and time instant in the UNiX format, such as yyyy-mm-dd hh:mm:ss[.f...]. For example, TIMESTAMP '2008-09-15 03:04:05.324'. This format uses the session time zone.

String

Name Type Comments
STRING Constant A string literal enclosed in single or double quotes
decimal(precision: number, scale?: number) Function precision is the total number of digits. scale (optional) is the number of digits in fractional part with a default of 0. For example, use these type definitions: decimal(11,5), decimal(15)
char(length: number) Function Fixed length character data, with a specified length between 1 and 255, such as char(10)
varchar(length: number) Function Variable length character data, with a specified length between 1 and 65535, such as varchar(10)

Miscellaneous

Name Type Comments
BOOLEAN Constant Values are true and false
BINARY Constant Value is in binary

Complex

Name Type Comments
array(itemType: Type) Function An array of some other type
map(keyType: Type, valueType: Type) Function A map of some primitive key type to any value type
struct(collumns: Column[]) Function Nested structure containing individually named and typed collumns

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

aws-cdk.aws-glue-1.119.0.tar.gz (254.9 kB view hashes)

Uploaded Source

Built Distribution

aws_cdk.aws_glue-1.119.0-py3-none-any.whl (252.9 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page