Sumo Logic AWS CloudFormation Testing Framework

The framework can be used to test the full life cycle of a AWS CloudFormation template from Validation, Deployment, Post Deployment and Clean Up.

The framework performs below life cycle events for an AWS CloudFormation template:

1. Pre Deployment Event - Perform pre validation which include CFN Linting and CFN Nag to check for validation issue and security issue.
2. Deploy Event - Perform deployment of a CloudFormation template.
3. Post Deployment Event - Perform post deployment validation like Resource, Outputs and parameters checks.
4. Clean Up Event - Perform clean up of CloudFormation stack.

Pre Requisite

Before using the testing framework,

1. Install AWS CLI on the machine.
2. Install CFN NAG in the machine. To install CFN NAG, please visit.

Installation

Framework can be installed as a python package using pip3 install sumologic-cfn-tester

Usage

To test the CloudFormation template, run command sumocfntester -f <Test File>

An Extra flag -d true can be used to enable debug logging.

Test File

Framework requires a Test File (JSON OR YAML) format, which contains test cases. Below is an example for a valid Test File. Comments are added for each field.

---
# Global Configuration
Global:
  TemplatePath: "../../templates/ParentTemplate.yaml" # Path of the template
  TestProjectName: BasicTestProject # Test Project Name
  ParallelTestsRun: 5 # Number of parallel Tests run. Number of stacks will be deployed parallel on AWS.
  GlobalParameters: # Key value pairs for the CloudFormation template parameter. Key = Parameter Name, value = Parameter value.
    SumoDeployment: us1
    SumoAccessID: Parameter_value_2 # You can also provide env variables like ${ENV_1}
    SumoAccessKey: Parameter_value_1
    SumoOrganizationId: Parameter_value_2
    RemoveSumoResourcesOnDeleteStack: 'true'
# Tests Configuration
Tests:
  - TestName: Do_Not_Install_Nested_Stack # Test Case Number 1
    Regions: # List of regions where you want to deploy the stack.
      - ap-south-1
    Parameters: # Contains Path Or Values. Values: Key value pairs for the CloudFormation template parameter. Key = Parameter Name, value = Parameter value.
      Values:
        InstallApp: 'No'
        CreateCloudWatchMetricsSource: 'No'
    Skip: false # True if need to skip the test case.
    Assertions: # All Assertions for post deployment validation. Nested Resources can be provided separated by '.'.
      - AssertType: ResourceExistence # ResourceExistence validation. Can have list of Resources in Assert object.
        Assert:
          Resources:
            - SumoLogicHelperRole
            - SumoLogicHelperFunction
            - SumoRole
            - sumoNestedAppStack.SumoRole # Shows Nested Stack Resource logical ID. sumoNestedAppStack -> Resource Logical ID of Stack in Parent stack. SumoRole = A Resource in Nested Stack.
      - AssertType: OutputsCheck # OutputsCheck validation. Can have list of Outputs in Assert object.
        Assert:
          Outputs:
            - LambdaHelperARN
            - LambdaRoleARN
            - SumoRoleARN
            - sumoNestedAppStack.SumoRoleARN
      - AssertType: ParameterCheck # ParameterCheck validation. Can have Dictionary of Parameters in Assert object.
        Assert:
          sumoNestedAppStack: # Resource logical ID. Key value pairs for the CloudFormation template parameter. Key = Parameter Name, value = Parameter value.
            ParentStackName: MasterTemplate
            InstallApp: 'Yes'
          sumoNestedAppStack.sumoNestedAppStack: # Resource logical ID. Key value pairs for the CloudFormation template parameter. Key = Parameter Name, value = Parameter value.
            ParentStackName: DoNotInstall
            InstallApp: 'Yes'
  - TestName: Install_Nested_Stack # Test Case Number 2
    Regions:
      - ap-south-1
    Parameters:
      Path: Install_Nested_Stack.yaml # Contains Path Or Values. Path: Path of the parameter file.
    Skip: false
    Assertions:
      - AssertType: ResourceExistence
        Assert:
          Resources:
            - SumoLogicHelperRole
            - SumoLogicHelperFunction
            - SumoRole
            - sumoNestedAppStack.SumoRole
            - sumoNestedAppStack
            - sumoNestedAppStack.sumoNestedAppStack
      - AssertType: OutputsCheck
        Assert:
          Outputs:
            - LambdaHelperARN
            - LambdaRoleARN
            - SumoRoleARN
            - sumoNestedAppStack
            - sumoNestedAppStack.SumoRoleARN
            - sumoNestedAppStack.sumoNestedAppStack
      - AssertType: ParameterCheck
        Assert:
          sumoNestedAppStack:
            ParentStackName: MasterTemplate
            InstallApp: 'Yes'
          sumoNestedAppStack.sumoNestedAppStack:
            ParentStackName: DoNotInstall
            InstallApp: 'Yes'

Process Flow

Below Steps explains all the processes a CF template goes through.

Pre Deployment

For Pre Deployment testing (Validation done on CF template before deploying it on AWS), we use below third party packages.

A. CFN PYTHON LINT

• Install the dependency using pip install cfn-lint.
• Helps to perform some basic validation on CF template to check the resources, mapping, parameters.
• It also checks for conditional dependencies within CF templates.
• For more details on all rules, try running cfn-lint -l after installing dependency.

B. CFN NAG

• The dependency require ruby to installed on the machine.
• The dependency can be installed using gem install cfn-nag
• Helps to check basic rules like S3 Bucket policy, Wild cards checks etc.

Deployment

Deployment process deploys the CloudFormation template to AWS account with the mentioned regions.

Post Deployment

Post Deployment process check for below validation on the deployed CloudFormation stack.

A. ResourceExistence - The validation checks for all mentioned resources in your CloudFormation stack and corresponding nested stacks. B. OutputsCheck - The validation checks for all mentioned resources in your CloudFormation stack and corresponding nested stacks. C. ParameterCheck - The validation checks for all mentioned resources in your CloudFormation stack and corresponding nested stacks.

Clean Up

Clean Up process deletes the deployed stack from AWS account.

Report Generation

A report will be generated for each test file with the status of the test cases within the test files.