Skip to main content
https://travis-ci.org/ImmobilienScout24/pybuilder_aws_plugin.svg?branch=master https://coveralls.io/repos/ImmobilienScout24/pybuilder_aws_plugin/badge.svg?branch=master&service=github https://badge.fury.io/py/pybuilder_aws_plugin.svg

pybuilder_aws_plugin

PyBuilder plugin to simplify building projects for Amazon Web Services. The following use cases are supported:

Incompatible Changes

>161 (March 2016) Upload and Release are Separate Steps

Till up to version 161 upload_zip_to_s3 and upload_cfn_to_s3 tasks would upload the files to S3 both under a versioned (v123) path and under a latest path element. This behavior prevents testing the new version before releasing it under the latest path.

Since the task name “upload” does not imply “release” and since we believe in Test Driven Development we decided to break backwards compatibility in this case.

From version 162 and onward the “upload” tasks will only upload files to S3 under a versioned path. We provide two new tasks lambda_release and cfn_release to explicitly copy the files from the versioned path to the latest path.

We apologize for the inconvenience and hope that this change will simplify your integration tests.

Usage

Add the following plugin dependency to your build.py (will install directly from PyPi):

use_plugin('pypi:pybuilder_aws_plugin')

After this you have the following additional tasks, which are explained below:

  • package_lambda_code

  • upload_zip_to_s3

  • upload_cfn_to_s3

  • lambda_release

  • cfn_release

  • upload_custom_resource

  • release_custom_resource

@Task: package_lambda_code

This task assembles the Zip-file (a.k.a. the lambda-zip) which will be uploaded to S3 with the task upload_zip_to_s3. This task consists of the following steps:

Add all dependencies

Install every entry in build.py, that is specified by using project.depends_on(), into a temporary directory via pip install -t. These will be included in the resulting lambda-zip. Set the project property install_dependencies_index_url to use a custom index url (e.g. an internal PYPI server).

Note: This excludes boto and boto3 as they are included in AWS lambda dependencies by default

Add all own modules

All modules which are found in src/main/python/ are copied directly into the lambda-zip.

Add all script files

The content of the scripts folder (src/main/scripts) in a PyBuilder project is normally intended to be placed in /usr/bin. This plugin assumes this directory contains scripts including the lambda handler functions. Therefore all files under this folder are copied directly to the root directory (/) of the lambda-zip.

Pack everything into the Zip-file

All these files are packed as a Zip-file that complies with the Lambda specification.

@Task: upload_zip_to_s3

This task uploads the generated zip to an S3 bucket. The bucket name is set in build.py:

project.set_property('bucket_name', 'my_lambda_bucket')

The default acl for zips to be uploaded is bucket-owner-full-control. But if you need another acl you can overwrite this as follows in build.py:

project.set_property('lambda_file_access_control', '<acl>')

Possible acl values are:

  • private

  • public-read

  • public-read-write

  • authenticated-read

  • bucket-owner-read

  • bucket-owner-full-control

Furthermore, the plugin assumes that you already have a shell with enabled AWS access (exported keys or .boto or …). afp-cli is a tool to provide temporary credentials for shell users.

The uploaded files will be placed in a directory with the version number like: v123/projectname.zip.

Use the property bucket_prefix to add a prefix to the uploaded files. For example:

project.set_property('bucket_prefix', 'my_lambda/')

This will upload the zip-file to the following key: my_lambda/v123/projectname.zip

On TeamCity you can enable setting a TeamCity build parameter with the key of the uploaded zip-file:

After uploading the zip-file to S3 the plugin will emit a

##teamcity[setParameter name='my_tc_parameter' value='my_lambda/v123/project-name.zip']

line which TeamCity can parse. You can then use the value in other build steps.

@Task: upload_cfn_to_s3

NOTE: This task is available for Python 2.7 and up, due to cfn-sphere dependencies not being available for Python 2.6.

This task converts and uploads the CFN-Sphere template YAML files as JSON to a S3 bucket. Set the bucket name in build.py:

project.set_property('bucket_name', 'my_template_bucket')

Define the CFN templates to upload via a list of tuples in the template_files property:

project.set_property('template_files',
    [
        ('path1','filename1.yaml'),
        ('path2','filename2.yaml'),
        ...
    ])

The uploaded files will be placed in a directory with the version number:

  • v123/filename1.json

  • v123/filename2.json

Use the property template_key_prefix to add a prefix to the uploaded files. For example:

project.set_property('template_key_prefix', 'my_template/')

This will upload the files to the following files:

  • my_template/v123/filename1.json

  • my_template/v123/filename2.json

The ACL for the JSON files is bucket-owner-full-control. Set another ACL in build.py:

project.set_property('template_file_access_control', '<acl>')

Possible acl values are:

  • private

  • public-read

  • public-read-write

  • authenticated-read

  • bucket-owner-read

  • bucket-owner-full-control

@Task: lambda_release, cfn_release

These tasks copy the lambda-zip or CFN template files from the versioned path to version independant path named latest. For Example:

  • my_lambda/v123/my-project.zip is copied to my_lambda/latest/my-project.zip

  • my_templates/v123/my-cfn.json is copied to my_templates/latest/my-cfn.json

This provides a simple release mechanism that follows the “latest greatest” principle. Users can rely on the files under latest to be the latest tested version.

@Task: upload_custom_resource, release_custom_resource

For CloudFormation custom resources backed by a Lambda function these two tasks provide convenience wrappers to implement an “Update - Test - Release” process:

#!/bin/bash
set -e
pyb upload_custom_resource
./run-integration-test.py
pyb release_custom_resource

The upload_custom_resource task bundles the upload_zip_to_s3 and the upload_cfn_to_s3 task. It is strongly recmomended to not use a bucket_prefix in order to keep the lambda-zip and CFN templates in the same direcory on S3.

Licence

Copyright 2015,2016 Immobilien Scout GmbH

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Release files for pybuilder_aws_plugin 183

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pybuilder_aws_plugin 183
File Size Uploaded
pybuilder_aws_plugin-183.tar.gz 7.7 kB Details

Release files / pybuilder_aws_plugin-183.tar.gz

Download URL pybuilder_aws_plugin-183.tar.gz
Size 7.7 kB
Tags Source
SHA-256 checksum
How to use checksums
36dd353275aac2e4ddf1f914b330ebc6add645616ebffa6bf668d1eb43281a81
BLAKE2b-256 checksum
How to use checksums
7b8bca5bb24619fe8019a3d7fa9ba29e50bd20dbd7ab2a519659075c66ea0fec
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release history Release notifications | RSS feed

This release

183 This release

1 release file

181

1 release file

176

1 release file

173

1 release file

172

1 release file

170

1 release file

161

1 release file

158

1 release file

157

1 release file

156

1 release file

155

1 release file

154

1 release file

151

1 release file

148

1 release file

144

1 release file

129

1 release file

121

1 release file

120

1 release file

119

1 release file

118

1 release file

117

1 release file

116

1 release file

115

1 release file

1

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page