AWS Config Rule Integration TesTER
Project description
critter
- AWS Config Rule Integration TesTER
critter
enables AWS Config rule integration testing. Use it to validate that AWS Config rules evaluate resources as expected.
Customers use AWS Config rules to evaluate their AWS resources against their own unique compliance and governance requirements. critter
is a command line tool that enables a continuous integration workflow to validate that Config rules evaluate resources as expected. This is essential to guaranteeing compliance within AWS accounts, especially when you consider the potential impact of unexpected behavior from Config rule auto-remediations.
This project is in MVP phase - it is functional but the api may change significantly before release 1.0
.
Overview
Usage of critter
within the Config rule development workflow looks like this:
- Developer writes Config custom rule evaluation Lambda function code that evaluates AWS resources.
- Example Config custom rules Python code
critter
can also test Config managed rules
- Developer deploys their Config rule and Lambda function.
- CloudFormation, the RDK (Rule Development Kit), or Terraform can be used to deploy Config rules.
- Example Config rules CloudFormation templates
- Developer creates a CloudFormation template that defines test resources to be evaluated by the Config rule
- Example
critter
test CloudFormation templates - The template declares a few outputs that will be used by
critter
Resources: CompliantResource1: # ... CompliantResource2: # ... NonCompliantResource1: # ... NonCompliantResource2: # ... Outputs: ConfigRuleName: Value: my-custom-config-rule CompliantResourceIds: Value: !Sub "${CompliantResource1.Id},${CompliantResource2.Id}" NonCompliantResourceIds: Value: !Sub "${NonCompliantResource1.Id},${NonCompliantResource2.Id}"
- Example
- Developer triggers
critter
from the command line (or from within a CI/CD system).critter
then ...- Deploys the
critter
test CloudFormation stack - Waits for the resources to be evaluated by the Config rule
- Validates that resources declared in output
CompliantResourceIds
were evaluated asCOMPLIANT
and resources declared in outputNonCompliantResourceIds
were evaluated asNON_COMPLIANT
- Deletes the
critter
test CloudFormation stack
- Deploys the
- Developer is confident their Config rule will evaluate resources as expected!
Install
Install with Python pip
from the GitHub project using git + ssh:
pip install git+ssh://git@github.com:awslabs/critter.git
After critter
is installed, you can verify the installation by showing the help: critter -h
.
Usage
- Deploy an AWS Config rule. The rule can be a managed rule or a custom rule that utilizes an AWS Lambda function for resource valuation.
- Create a
critter
test CloudFormation template (i.e.my-test-template.yml
) that deploys at least one resource you expect the rule to evaluate asCOMPLIANT
and at least one resource you expect the rule to evaluate asNON_COMPLIANT
. - Declare the following
Outputs
in thecritter
test CloudFormation template (documentation below):ConfigRuleName
: The name of the Config rule to testCompliantResourceIds
: A comma separated list of one or more AWS Config resource IDs expected to be evaluated asCOMPLIANT
.NonCompliantResourceIds
: A comma separated list of one or more AWS Config resource IDs expected to be evaluated asNON_COMPLIANT
.
critter TEMPLATE
(i.e.critter ./my-test-template.yml
) deploys thecritter
test CloudFormation stack and validates the resource evaluations.critter
AWS integration is configured with standardboto3
configuration (environment variables and the~/.aws/config
file).- The
critter
test CloudFormation stack will be deleted after testingOnSuccess
by default (i.e. if all tests pass). This behavior can be controlled with--delete-stack
.
Continuous Integration
To understand how critter
can be utilized in a Continuous Integration (CI) workflow to automatically test changes to AWS Config rules, see the AWS CodeBuild CI example in examples/ci-pipelines/aws-codebuild/
.
AWS Config Resource IDs
Most AWS resources have an id
attribute (or similar) that is used as the AWS Config resource ID. For EC2 instances, resource IDs are the EC2 instance IDs (i.e. i-111111111aaaaaaaa,i-222222222bbbbbbbb
). For VPC security groups, the resource ID is the security group ID (i.e. sg-333333333cccccccc
). For IAM roles, the resource ID is the role ID (i.e. AROAJI4AVVEXAMPLE
, which can be retrieved in a CloudFormation template using Fn::Sub '${MyIamRole.RoleId}'
).
Generally, resource IDs can be obtained using the CloudFormation intrinsic Ref
function with the resource's logical name. To confirm which attribute is used as a resource's Config resource ID, there are a few options:
- Browse resources in the AWS Config console
- Browse resources using the AWS CLI:
aws configservice list-discovered-resources --resource-type AWS::EC2::Instance
- See the
resourceId
field on example configuration items here.
critter
Test CloudFormation Stack Outputs
See example critter
test CloudFormation templates in the ./examples/test-stacks/
. Below is an explanation of each of the supported test stack outputs. Note that CloudFormation stack outputs are always strings.
ConfigRuleName
- AWS Config rule name to be tested. Be sure the rule is already deployed within the account and Region the
critter
test CloudFormation stack will be deployed to. - Example values:
"my-custom-config-rule"
- AWS Config rule name to be tested. Be sure the rule is already deployed within the account and Region the
CompliantResourceIds
- A comma separated list of one or more AWS Config resource IDs expected to be evaluated as
COMPLIANT
. At least one ofCompliantResourceIds
andNonCompliantResourceIds
must be output. - Typically generated with
!Sub "${CompliantResource1.Id},${CompliantResource2.Id}"
- Example values:
"i-11111111111111111"
"sg-22222222222222222,sg-33333333333333333"
- A comma separated list of one or more AWS Config resource IDs expected to be evaluated as
NonCompliantResourceIds
- A comma separated list of one or more AWS Config resource IDs expected to be evaluated as
NON_COMPLIANT
. At least one ofCompliantResourceIds
andNonCompliantResourceIds
must be output. - Typically generated with
!Sub "${NonCompliantResource1.Id},${NonCompliantResource2.Id}"
- Example values:
"sg-11111111111111111"
"i-22222222222222222,i-33333333333333333"
- A comma separated list of one or more AWS Config resource IDs expected to be evaluated as
DelayAfterDeploy
- Seconds to delay after
critter
test CloudFormation stack create or update. If the test stack is already deployed and no update occurs, the delay will be skipped. - Example values:
"60"
- Seconds to delay after
SkipWaitForResourceRecording
- Skip waiting for resources to be recorded in Config. Useful if Config rule is testing resources not natively supported by Config.
- Allowed values:
"True"
"False"
(default behavior)
CLI Options
Warning - This documentation may become out of date until the api stabilizes. Show the up to date help with critter -h
.
$ critter -h
usage: critter [-h] [--trigger-rule-evaluation] [--stack-name STACK-NAME] [--stack-tags '[{"Key": "TagKey", "Value": "TagValue"}, ...]']
[--capabilities CAPABILITY [CAPABILITY ...]] [--delete-stack {Always,OnSuccess,Never}]
TEMPLATE
critter - AWS Config Rule Integration TesTER
positional arguments:
TEMPLATE CloudFormation template(s) to test already deployed Config rule
optional arguments:
-h, --help show this help message and exit
--trigger-rule-evaluation
Trigger Config rule evaluation after CloudFormation stack deployment. Useful for periodic evaluation rules. Also ensures the rule
evaluation occured after stack deployment.
--stack-name STACK-NAME
CloudFormation stack name (default is generated from TEMPLATE file name)
--stack-tags '[{"Key": "TagKey", "Value": "TagValue"}, ...]'
Tags to associate with the CloudFormation stack formatted as a JSON string
--capabilities CAPABILITY [CAPABILITY ...]
CloudFormation capabilities needed to deploy the stack (i.e. CAPABILITY_IAM, CAPABILITY_NAMED_IAM)
--delete-stack {Always,OnSuccess,Never}
Test outcome that should trigger CloudFormation stack delete (default: OnSuccess)
Contributing and Security
See CONTRIBUTING for more information.
License
This project is licensed under the Apache-2.0 License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.