Constructs for deploying contents to S3 buckets
Project description
AWS S3 Deployment Construct Library
---Status: Experimental
This library allows populating an S3 bucket with the contents of .zip files from other S3 buckets or from local disk.
The following example defines a publicly accessible S3 bucket with web hosting enabled and populates it from a local directory on disk.
# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
website_bucket = s3.Bucket(self, "WebsiteBucket",
website_index_document="index.html",
public_read_access=True
)
s3deploy.BucketDeployment(self, "DeployWebsite",
sources=[s3deploy.Source.asset("./website-dist")],
destination_bucket=website_bucket,
destination_key_prefix="web/static"
)
This is what happens under the hood:
- When this stack is deployed (either via
cdk deploy
or via CI/CD), the contents of the localwebsite-dist
directory will be archived and uploaded to an intermediary assets bucket. If there is more than one source, they will be individually uploaded. - The
BucketDeployment
construct synthesizes a custom CloudFormation resource of typeCustom::CDKBucketDeployment
into the template. The source bucket/key is set to point to the assets bucket. - The custom resource downloads the .zip archive, extracts it and issues
aws s3 sync --delete
against the destination bucket (in this casewebsiteBucket
). If there is more than one source, the sources will be downloaded and merged pre-deployment at this step.
Supported sources
The following source types are supported for bucket deployments:
- Local .zip file:
s3deploy.Source.asset('/path/to/local/file.zip')
- Local directory:
s3deploy.Source.asset('/path/to/local/directory')
- Another bucket:
s3deploy.Source.bucket(bucket, zipObjectKey)
To create a source from a single file, you can pass AssetOptions
to exclude
all but a single file:
- Single file:
s3deploy.Source.asset('/path/to/local/directory', { exclude: ['**', '!onlyThisFile.txt'] })
IMPORTANT The aws-s3-deployment
module is only intended to be used with
zip files from trusted sources. Directories bundled by the CDK CLI (by using
Source.asset()
on a directory) are safe. If you are using Source.asset()
or
Source.bucket()
to reference an existing zip file, make sure you trust the
file you are referencing. Zips from untrusted sources might be able to execute
arbitrary code in the Lambda Function used by this module, and use its permissions
to read or write unexpected files in the S3 bucket.
Retain on Delete
By default, the contents of the destination bucket will not be deleted when the
BucketDeployment
resource is removed from the stack or when the destination is
changed. You can use the option retainOnDelete: false
to disable this behavior,
in which case the contents will be deleted.
Configuring this has a few implications you should be aware of:
-
Logical ID Changes
Changing the logical ID of the
BucketDeployment
construct, without changing the destination (for example due to refactoring, or intentional ID change) will result in the deletion of the objects. This is because CloudFormation will first create the new resource, which will have no affect, followed by a deletion of the old resource, which will cause a deletion of the objects, since the destination hasn't changed, andretainOnDelete
isfalse
. -
Destination Changes
When the destination bucket or prefix is changed, all files in the previous destination will first be deleted and then uploaded to the new destination location. This could have availability implications on your users.
General Recommendations
Shared Bucket
If the destination bucket is not dedicated to the specific BucketDeployment
construct (i.e shared by other entities),
we recommend to always configure the destinationKeyPrefix
property. This will prevent the deployment from
accidentally deleting data that wasn't uploaded by it.
Dedicated Bucket
If the destination bucket is dedicated, it might be reasonable to skip the prefix configuration,
in which case, we recommend to remove retainOnDelete: false
, and instead, configure the
autoDeleteObjects
property on the destination bucket. This will avoid the logical ID problem mentioned above.
Prune
By default, files in the destination bucket that don't exist in the source will be deleted
when the BucketDeployment
resource is created or updated. You can use the option prune: false
to disable
this behavior, in which case the files will not be deleted.
# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
s3deploy.BucketDeployment(self, "DeployMeWithoutDeletingFilesOnDestination",
sources=[s3deploy.Source.asset(path.join(__dirname, "my-website"))],
destination_bucket=destination_bucket,
prune=False
)
This option also enables you to specify multiple bucket deployments for the same destination bucket & prefix, each with its own characteristics. For example, you can set different cache-control headers based on file extensions:
# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
BucketDeployment(self, "BucketDeployment",
sources=[Source.asset("./website", exclude=["index.html"])],
destination_bucket=bucket,
cache_control=[CacheControl.from_string("max-age=31536000,public,immutable")],
prune=False
)
BucketDeployment(self, "HTMLBucketDeployment",
sources=[Source.asset("./website", exclude=["*", "!index.html"])],
destination_bucket=bucket,
cache_control=[CacheControl.from_string("max-age=0,no-cache,no-store,must-revalidate")],
prune=False
)
Exclude and Include Filters
There are two points at which filters are evaluated in a deployment: asset bundling and the actual deployment. If you simply want to exclude files in the asset bundling process, you should leverage the exclude
property of AssetOptions
when defining your source:
# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
BucketDeployment(self, "HTMLBucketDeployment",
sources=[Source.asset("./website", exclude=["*", "!index.html"])],
destination_bucket=bucket
)
If you want to specify filters to be used in the deployment process, you can use the exclude
and include
filters on BucketDeployment
. If excluded, these files will not be deployed to the destination bucket. In addition, if the file already exists in the destination bucket, it will not be deleted if you are using the prune
option:
# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
s3deploy.BucketDeployment(self, "DeployButExcludeSpecificFiles",
sources=[s3deploy.Source.asset(path.join(__dirname, "my-website"))],
destination_bucket=destination_bucket,
exclude=["*.txt"]
)
These filters follow the same format that is used for the AWS CLI. See the CLI documentation for information on Using Include and Exclude Filters.
Objects metadata
You can specify metadata to be set on all the objects in your deployment.
There are 2 types of metadata in S3: system-defined metadata and user-defined metadata.
System-defined metadata have a special purpose, for example cache-control defines how long to keep an object cached.
User-defined metadata are not used by S3 and keys always begin with x-amz-meta-
(this prefix is added automatically).
System defined metadata keys include the following:
- cache-control (
--cache-control
inaws s3 sync
) - content-disposition (
--content-disposition
inaws s3 sync
) - content-encoding (
--content-encoding
inaws s3 sync
) - content-language (
--content-language
inaws s3 sync
) - content-type (
--content-type
inaws s3 sync
) - expires (
--expires
inaws s3 sync
) - x-amz-storage-class (
--storage-class
inaws s3 sync
) - x-amz-website-redirect-location (
--website-redirect
inaws s3 sync
) - x-amz-server-side-encryption (
--sse
inaws s3 sync
) - x-amz-server-side-encryption-aws-kms-key-id (
--sse-kms-key-id
inaws s3 sync
) - x-amz-server-side-encryption-customer-algorithm (
--sse-c-copy-source
inaws s3 sync
) - x-amz-acl (
--acl
inaws s3 sync
)
You can find more information about system defined metadata keys in
S3 PutObject documentation
and aws s3 sync
documentation.
# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
website_bucket = s3.Bucket(self, "WebsiteBucket",
website_index_document="index.html",
public_read_access=True
)
s3deploy.BucketDeployment(self, "DeployWebsite",
sources=[s3deploy.Source.asset("./website-dist")],
destination_bucket=website_bucket,
destination_key_prefix="web/static", # optional prefix in destination bucket
metadata={"A": "1", "b": "2"}, # user-defined metadata
# system-defined metadata
content_type="text/html",
content_language="en",
storage_class=StorageClass.INTELLIGENT_TIERING,
server_side_encryption=ServerSideEncryption.AES_256,
cache_control=[CacheControl.set_public(), CacheControl.max_age(cdk.Duration.hours(1))],
access_control=s3.BucketAccessControl.BUCKET_OWNER_FULL_CONTROL
)
CloudFront Invalidation
You can provide a CloudFront distribution and optional paths to invalidate after the bucket deployment finishes.
# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
import aws_cdk.aws_cloudfront as cloudfront
import aws_cdk.aws_cloudfront_origins as origins
bucket = s3.Bucket(self, "Destination")
# Handles buckets whether or not they are configured for website hosting.
distribution = cloudfront.Distribution(self, "Distribution",
default_behavior=BehaviorOptions(origin=origins.S3Origin(bucket))
)
s3deploy.BucketDeployment(self, "DeployWithInvalidation",
sources=[s3deploy.Source.asset("./website-dist")],
destination_bucket=bucket,
distribution=distribution,
distribution_paths=["/images/*.png"]
)
Memory Limit
The default memory limit for the deployment resource is 128MiB. If you need to
copy larger files, you can use the memoryLimit
configuration to specify the
size of the AWS Lambda resource handler.
NOTE: a new AWS Lambda handler will be created in your stack for each memory limit configuration.
EFS Support
If your workflow needs more disk space than default (512 MB) disk space, you may attach an EFS storage to underlying
lambda function. To Enable EFS support set efs
and vpc
props for BucketDeployment.
Check sample usage below. Please note that creating VPC inline may cause stack deletion failures. It is shown as below for simplicity. To avoid such condition, keep your network infra (VPC) in a separate stack and pass as props.
# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826
s3deploy.BucketDeployment(self, "DeployMeWithEfsStorage",
sources=[s3deploy.Source.asset(path.join(__dirname, "my-website"))],
destination_bucket=destination_bucket,
destination_key_prefix="efs/",
use_efs=True,
vpc=ec2.Vpc(self, "Vpc"),
retain_on_delete=False
)
Notes
- This library uses an AWS CloudFormation custom resource which about 10MiB in size. The code of this resource is bundled with this library.
- AWS Lambda execution time is limited to 15min. This limits the amount of data which can be deployed into the bucket by this timeout.
- When the
BucketDeployment
is removed from the stack, the contents are retained in the destination bucket (#952). - Bucket deployment only happens during stack create/update. This means that if you wish to update the contents of the destination, you will need to change the source s3 key (or bucket), so that the resource will be updated. This is inline with best practices. If you use local disk assets, this will happen automatically whenever you modify the asset, since the S3 key is based on a hash of the asset contents.
Development
The custom resource is implemented in Python 3.6 in order to be able to leverage
the AWS CLI for "aws s3 sync". The code is under lib/lambda
and
unit tests are under test/lambda
.
This package requires Python 3.6 during build time in order to create the custom resource Lambda bundle and test it. It also relies on a few bash scripts, so might be tricky to build on Windows.
Roadmap
- Support "blue/green" deployments (#954)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for aws-cdk.aws-s3-deployment-1.130.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5629c69a15e39f8425661e9a4d1f6dfeea38faeed9967752f63224c1dc3c67b5 |
|
MD5 | 5131b93d22cfa5ee13aae84eb707e1de |
|
BLAKE2b-256 | 9b6191328a390117b7058a7e2e04e9af9542a908f545106336902a5d89fa7fad |
Hashes for aws_cdk.aws_s3_deployment-1.130.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 55717c7a3b0c11fcf2564aadda5234b0b0b539a27a80314a5a0e4b8e04f0473f |
|
MD5 | e52d10495dfd97177a0efec522af78d8 |
|
BLAKE2b-256 | 38a7f5bf922fdc573a2c3c6b34ad303de32cd412d67b5106585934e6f89e4db5 |