Skip to main content

AWS Config Rule Integration TesTER

Project description

critter - AWS Config Rule Integration TesTER

critter enables AWS Config rule integration testing. Use it to validate that AWS Config rules evaluate resources as expected.

Customers use AWS Config rules to evaluate their AWS resources against their own unique compliance and governance requirements. critter is a command line tool that enables a continuous integration workflow to validate that Config rules evaluate resources as expected. This is essential to guaranteeing compliance within AWS accounts, especially when you consider the potential impact of unexpected behavior from Config rule auto-remediations.

This project is in MVP phase - it is functional but the api may change significantly before release 1.0.

Overview

Usage of critter within the Config rule development workflow looks like this:

  1. Developer writes Config custom rule evaluation Lambda function code that evaluates AWS resources.
  2. Developer deploys their Config rule and Lambda function.
  3. Developer creates a CloudFormation template that defines test resources to be evaluated by the Config rule
    • Example critter test CloudFormation templates
    • The template declares a few outputs that will be used by critter
      Resources:
        CompliantResource1:
          # ...
        CompliantResource2:
          # ...
        NonCompliantResource1:
          # ...
        NonCompliantResource2:
          # ...
      
      Outputs:
        ConfigRuleName:
          Value: my-custom-config-rule
        CompliantResourceIds:
          Value: !Sub "${CompliantResource1.Id},${CompliantResource2.Id}"
        NonCompliantResourceIds:
          Value: !Sub "${NonCompliantResource1.Id},${NonCompliantResource2.Id}"
      
  4. Developer triggers critter from the command line (or from within a CI/CD system). critter then ...
    1. Deploys the critter test CloudFormation stack
    2. Waits for the resources to be evaluated by the Config rule
    3. Validates that resources declared in output CompliantResourceIds were evaluated as COMPLIANT and resources declared in output NonCompliantResourceIds were evaluated as NON_COMPLIANT
    4. Deletes the critter test CloudFormation stack
  5. Developer is confident their Config rule will evaluate resources as expected!

Install

Install with Python pip from the GitHub project using git + ssh:

pip install git+ssh://git@github.com:awslabs/critter.git

After critter is installed, you can verify the installation by showing the help: critter -h.

Usage

  1. Deploy an AWS Config rule. The rule can be a managed rule or a custom rule that utilizes an AWS Lambda function for resource valuation.
  2. Create a critter test CloudFormation template (i.e. my-test-template.yml) that deploys at least one resource you expect the rule to evaluate as COMPLIANT and at least one resource you expect the rule to evaluate as NON_COMPLIANT.
  3. Declare the following Outputs in the critter test CloudFormation template (documentation below):
    1. ConfigRuleName: The name of the Config rule to test
    2. CompliantResourceIds: A comma separated list of one or more AWS Config resource IDs expected to be evaluated as COMPLIANT.
    3. NonCompliantResourceIds: A comma separated list of one or more AWS Config resource IDs expected to be evaluated as NON_COMPLIANT.
  4. critter TEMPLATE (i.e. critter ./my-test-template.yml) deploys the critter test CloudFormation stack and validates the resource evaluations.

Continuous Integration

To understand how critter can be utilized in a Continuous Integration (CI) workflow to automatically test changes to AWS Config rules, see the AWS CodeBuild CI example in examples/ci-pipelines/aws-codebuild/.

AWS Config Resource IDs

Most AWS resources have an id attribute (or similar) that is used as the AWS Config resource ID. For EC2 instances, resource IDs are the EC2 instance IDs (i.e. i-111111111aaaaaaaa,i-222222222bbbbbbbb). For VPC security groups, the resource ID is the security group ID (i.e. sg-333333333cccccccc). For IAM roles, the resource ID is the role ID (i.e. AROAJI4AVVEXAMPLE, which can be retrieved in a CloudFormation template using Fn::Sub '${MyIamRole.RoleId}').

Generally, resource IDs can be obtained using the CloudFormation intrinsic Ref function with the resource's logical name. To confirm which attribute is used as a resource's Config resource ID, there are a few options:

critter Test CloudFormation Stack Outputs

See example critter test CloudFormation templates in the ./examples/test-stacks/. Below is an explanation of each of the supported test stack outputs. Note that CloudFormation stack outputs are always strings.

  • ConfigRuleName
    • AWS Config rule name to be tested. Be sure the rule is already deployed within the account and Region the critter test CloudFormation stack will be deployed to.
    • Example values:
      • "my-custom-config-rule"
  • CompliantResourceIds
    • A comma separated list of one or more AWS Config resource IDs expected to be evaluated as COMPLIANT. At least one of CompliantResourceIds and NonCompliantResourceIds must be output.
    • Typically generated with !Sub "${CompliantResource1.Id},${CompliantResource2.Id}"
    • Example values:
      • "i-11111111111111111"
      • "sg-22222222222222222,sg-33333333333333333"
  • NonCompliantResourceIds
    • A comma separated list of one or more AWS Config resource IDs expected to be evaluated as NON_COMPLIANT. At least one of CompliantResourceIds and NonCompliantResourceIds must be output.
    • Typically generated with !Sub "${NonCompliantResource1.Id},${NonCompliantResource2.Id}"
    • Example values:
      • "sg-11111111111111111"
      • "i-22222222222222222,i-33333333333333333"
  • DelayAfterDeploy
    • Seconds to delay after critter test CloudFormation stack create or update. If the test stack is already deployed and no update occurs, the delay will be skipped.
    • Example values:
      • "60"
  • SkipWaitForResourceRecording
    • Skip waiting for resources to be recorded in Config. Useful if Config rule is testing resources not natively supported by Config.
    • Allowed values:
      • "True"
      • "False" (default behavior)

CLI Options

Warning - This documentation may become out of date until the api stabilizes. Show the up to date help with critter -h.

$ critter -h
usage: critter [-h] [--trigger-rule-evaluation] [--stack-name STACK-NAME] [--stack-tags '[{"Key": "TagKey", "Value": "TagValue"}, ...]']
               [--capabilities CAPABILITY [CAPABILITY ...]] [--delete-stack {Always,OnSuccess,Never}]
               TEMPLATE

critter - AWS Config Rule Integration TesTER

positional arguments:
  TEMPLATE              CloudFormation template(s) to test already deployed Config rule

optional arguments:
  -h, --help            show this help message and exit
  --trigger-rule-evaluation
                        Trigger Config rule evaluation after CloudFormation stack deployment. Useful for periodic evaluation rules. Also ensures the rule
                        evaluation occured after stack deployment.
  --stack-name STACK-NAME
                        CloudFormation stack name (default is generated from TEMPLATE file name)
  --stack-tags '[{"Key": "TagKey", "Value": "TagValue"}, ...]'
                        Tags to associate with the CloudFormation stack formatted as a JSON string
  --capabilities CAPABILITY [CAPABILITY ...]
                        CloudFormation capabilities needed to deploy the stack (i.e. CAPABILITY_IAM, CAPABILITY_NAMED_IAM)
  --delete-stack {Always,OnSuccess,Never}
                        Test outcome that should trigger CloudFormation stack delete (default: OnSuccess)

Contributing and Security

See CONTRIBUTING for more information.

License

This project is licensed under the Apache-2.0 License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

critter-0.1.3.tar.gz (17.4 kB view details)

Uploaded Source

Built Distribution

critter-0.1.3-py3-none-any.whl (15.1 kB view details)

Uploaded Python 3

File details

Details for the file critter-0.1.3.tar.gz.

File metadata

  • Download URL: critter-0.1.3.tar.gz
  • Upload date:
  • Size: 17.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7

File hashes

Hashes for critter-0.1.3.tar.gz
Algorithm Hash digest
SHA256 f1cfa6c17595bb60bcf578bc4cbd767c2f0dba100cbf3c9541ce0263bf4be1a1
MD5 0823be0db93d549928f94055da597239
BLAKE2b-256 f9778ff7eff9cbbc2cf226fa78a3c01609c14997593718da1ab3042e6ce256ce

See more details on using hashes here.

File details

Details for the file critter-0.1.3-py3-none-any.whl.

File metadata

  • Download URL: critter-0.1.3-py3-none-any.whl
  • Upload date:
  • Size: 15.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.10.1 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.9.7

File hashes

Hashes for critter-0.1.3-py3-none-any.whl
Algorithm Hash digest
SHA256 2a7ce7f2e442c74a9cb7bcd63c8bfc60f591e9f857ec42f41ac8c96985dc99a7
MD5 b65098850df6e6b2fae668c16a9d06e3
BLAKE2b-256 dbf4acfe42610fb56c4701bef38a2c134a791f2773120be8aafed16cc8ad0e50

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page