PyBuilder plugin to handle AWS functionality
PyBuilder plugin to simplify building projects for Amazon Web Services. The following use cases are supported:
- Packaging Python code for Lambda and uploading the result to S3.
- Maintain CloudFormation templates in YAML and upload to S3. Conversion done with cfn-sphere.
- Deploy code and templates for a CloudFormation custom resource backed by a Lambda function.
>161 (March 2016) Upload and Release are Separate Steps
Till up to version 161 upload_zip_to_s3 and upload_cfn_to_s3 tasks would upload the files to S3 both under a versioned (v123) path and under a latest path element. This behavior prevents testing the new version before releasing it under the latest path.
Since the task name “upload” does not imply “release” and since we believe in Test Driven Development we decided to break backwards compatibility in this case.
From version 162 and onward the “upload” tasks will only upload files to S3 under a versioned path. We provide two new tasks lambda_release and cfn_release to explicitly copy the files from the versioned path to the latest path.
We apologize for the inconvenience and hope that this change will simplify your integration tests.
Add the following plugin dependency to your build.py (will install directly from PyPi):
After this you have the following additional tasks, which are explained below:
Add all dependencies
Install every entry in build.py, that is specified by using project.depends_on(), into a temporary directory via pip install -t. These will be included in the resulting lambda-zip. Set the project property install_dependencies_index_url to use a custom index url (e.g. an internal PYPI server).
Note: This excludes boto and boto3 as they are included in AWS lambda dependencies by default
Add all own modules
All modules which are found in src/main/python/ are copied directly into the lambda-zip.
Add all script files
The content of the scripts folder (src/main/scripts) in a PyBuilder project is normally intended to be placed in /usr/bin. This plugin assumes this directory contains scripts including the lambda handler functions. Therefore all files under this folder are copied directly to the root directory (/) of the lambda-zip.
Pack everything into the Zip-file
All these files are packed as a Zip-file that complies with the Lambda specification.
This task uploads the generated zip to an S3 bucket. The bucket name is set in build.py:
The default acl for zips to be uploaded is bucket-owner-full-control. But if you need another acl you can overwrite this as follows in build.py:
Possible acl values are:
Furthermore, the plugin assumes that you already have a shell with enabled AWS access (exported keys or .boto or …). afp-cli is a tool to provide temporary credentials for shell users.
The uploaded files will be placed in a directory with the version number like: v123/projectname.zip.
Use the property bucket_prefix to add a prefix to the uploaded files. For example:
This will upload the zip-file to the following key: my_lambda/v123/projectname.zip
On TeamCity you can enable setting a TeamCity build parameter with the key of the uploaded zip-file:
After uploading the zip-file to S3 the plugin will emit a
##teamcity[setParameter name='my_tc_parameter' value='my_lambda/v123/project-name.zip']
line which TeamCity can parse. You can then use the value in other build steps.
NOTE: This task is available for Python 2.7 and up, due to cfn-sphere dependencies not being available for Python 2.6.
Define the CFN templates to upload via a list of tuples in the template_files property:
project.set_property('template_files', [ ('path1','filename1.yaml'), ('path2','filename2.yaml'), ... ])
The uploaded files will be placed in a directory with the version number:
Use the property template_key_prefix to add a prefix to the uploaded files. For example:
This will upload the files to the following files:
The ACL for the JSON files is bucket-owner-full-control. Set another ACL in build.py:
Possible acl values are:
@Task: lambda_release, cfn_release
These tasks copy the lambda-zip or CFN template files from the versioned path to version independant path named latest. For Example:
- my_lambda/v123/my-project.zip is copied to my_lambda/latest/my-project.zip
- my_templates/v123/my-cfn.json is copied to my_templates/latest/my-cfn.json
This provides a simple release mechanism that follows the “latest greatest” principle. Users can rely on the files under latest to be the latest tested version.
@Task: upload_custom_resource, release_custom_resource
For CloudFormation custom resources backed by a Lambda function these two tasks provide convenience wrappers to implement an “Update - Test - Release” process:
#!/bin/bash set -e pyb upload_custom_resource ./run-integration-test.py pyb release_custom_resource
The upload_custom_resource task bundles the upload_zip_to_s3 and the upload_cfn_to_s3 task. It is strongly recmomended to not use a bucket_prefix in order to keep the lambda-zip and CFN templates in the same direcory on S3.
Copyright 2015,2016 Immobilien Scout GmbH
Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.