ssmbak 0.2.1

Event-driven SSM Param backups and point-in-time restore.

The AWS SSM Parameter Store is simple and great for AWS config bits, but SSM only preserves 100 versions and maintains no record of deletion.

To enable point-in-time restore, including deleted versions and entire recursive trees, we use an s3 bucket with versioning enabled as a backend.

This project includes all the pieces to both backup and restore SSM Params to a point in time.

• Backup: Eventbridge -> SQS -> Lambda -> S3
  • launch cloudformation stack from template with ssmbak-stack <name> create.
• Restore with either:
  • ssmbak restore cli, which uses
  • the well-tested library

from ssmbak.restore.actions import ParamPath
ParamPath.restore()

Quickstart

You'll need credentials that can create IAM resources with Cloudformation (to assign minimal permissions to the lambda role).

pip install ssmbak
ssmbak-stack <SSMBAK_STACKNAME> create

That's it. All new params will automatically be backed-up and available for ssmbak point-in-time restore via CLI or lib, like:

ssmbak preview /my/ssm/path/ 2024-06-15T17:56:58

CLI Tutorial

You'll need the awcli unless you want to point and click in the AWS management console to follow along.

[!WARNING] There are sleeps in between steps to give SQS -> Lambda time to process. If AWS is slow, you might have to wait longer.

There's an experimental script in tests/verify_cli_tutorial.sh that does the steps. Feel free to follow along with the tail command described at the end of the tutorial.

First, create the stack.

SSMBAK_STACKNAME=ssmbak
ssmbak-stack $SSMBAK_STACKNAME create

06/15/24 17:25:25   CREATE_IN_PROGRESS  ssmbak  AWS::CloudFormation::Stack  User Initiated
...
06/15/24 17:26:44   CREATE_COMPLETE  ssmbak  AWS::CloudFormation::Stack

Once the stack is up and new params are backed-up automatically, you can go through the following steps to give you a feel for how it works.

Create some params with value initial in /testyssmbak/ and /testyssmbak/deeper to show recursion. We'll also set key /testyssmbak to show the difference between keys and paths.

aws ssm put-parameter --name /testyssmbak --value initial --type String --overwrite
for i in $(seq 3)
do
aws ssm put-parameter --name /testyssmbak/$i --value initial --type String --overwrite
aws ssm put-parameter --name /testyssmbak/deeper/$i --value initial --type String --overwrite
done

Standard        1
Standard        1
Standard        1
Standard        1
Standard        1
Standard        1
Standard        1

Sleep a bit to give EventBridge some time to process the event, mark it (UTC), and sleep some more to give ssmbak some time to back them up.

sleep 120
IN_BETWEEN=`date -u +"%Y-%m-%dT%H:%M:%S"`
sleep 120

They're all set to inital.

aws ssm get-parameters-by-path --path /testyssmbak --recursive \
  | perl -ne '@h=split; print "$h[4] \t\t $h[6]\n";'

/testyssmbak/1 		 initial
/testyssmbak/2 		 initial
/testyssmbak/3 		 initial
/testyssmbak/deeper/1 		 initial
/testyssmbak/deeper/2 		 initial
/testyssmbak/deeper/3 		 initial

Update #2 for path and subpath:

aws ssm put-parameter --name /testyssmbak/2 --value UPDATED --type String --overwrite
aws ssm put-parameter --name /testyssmbak/deeper/2 --value UPDATED --type String --overwrite

Standard        2
Standard        2

Let's sleep a bit before marking the time. Then we see that #2 for each is set to UPDATED:

sleep 120
UPDATED_MARK=`date -u +"%Y-%m-%dT%H:%M:%S"`
aws ssm get-parameters-by-path --path /testyssmbak --recursive \
  | perl -ne '@h=split; print "$h[4] \t\t $h[6]\n";'

/testyssmbak/1 		 initial
/testyssmbak/2 		 UPDATED
/testyssmbak/3 		 initial
/testyssmbak/deeper/1 		 initial
/testyssmbak/deeper/2 		 UPDATED
/testyssmbak/deeper/3 		 initial

When we preview the IN_BETWEEN point-in-time, we see that everything was initial at that time.

[!NOTE] ParamPaths end with a slash, which is why key /testyssmbak doesn't show up in the previews.

ssmbak preview /testyssmbak/ $IN_BETWEEN --recursive

+-----------------------+---------+--------+---------------------------+
| Name                  | Value   | Type   | Modified                  |
+-----------------------+---------+--------+---------------------------+
| /testyssmbak/1        | initial | String | 2024-06-15 17:48:58+00:00 |
| /testyssmbak/2        | initial | String | 2024-06-15 17:49:00+00:00 |
| /testyssmbak/3        | initial | String | 2024-06-15 17:49:01+00:00 |
| /testyssmbak/deeper/1 | initial | String | 2024-06-15 17:48:59+00:00 |
| /testyssmbak/deeper/2 | initial | String | 2024-06-15 17:49:00+00:00 |
| /testyssmbak/deeper/3 | initial | String | 2024-06-15 17:49:02+00:00 |
+-----------------------+---------+--------+---------------------------+

Do the restore:

ssmbak restore /testyssmbak/ $IN_BETWEEN --recursive

+-----------------------+---------+--------+---------------------------+
| Name                  | Value   | Type   | Modified                  |
+-----------------------+---------+--------+---------------------------+
| /testyssmbak/1        | initial | String | 2024-06-15 17:48:58+00:00 |
| /testyssmbak/2        | initial | String | 2024-06-15 17:49:00+00:00 |
| /testyssmbak/3        | initial | String | 2024-06-15 17:49:01+00:00 |
| /testyssmbak/deeper/1 | initial | String | 2024-06-15 17:48:59+00:00 |
| /testyssmbak/deeper/2 | initial | String | 2024-06-15 17:49:00+00:00 |
| /testyssmbak/deeper/3 | initial | String | 2024-06-15 17:49:02+00:00 |
+-----------------------+---------+--------+---------------------------+

And now they're all back to initial:

aws ssm get-parameters-by-path --path /testyssmbak --recursive \
  | perl -ne '@h=split; print "$h[4] \t\t $h[6]\n";'

/testyssmbak/1 		 initial
/testyssmbak/2 		 initial
/testyssmbak/3 		 initial
/testyssmbak/deeper/1 		 initial
/testyssmbak/deeper/2 		 initial
/testyssmbak/deeper/3 		 initial

Let's say we made a mistake and want to revert one of the UPDATED keys:

ssmbak preview /testyssmbak/deeper/2 $UPDATED_MARK --recursive

+-----------------------+---------+--------+---------------------------+
| Name                  | Value   | Type   | Modified                  |
+-----------------------+---------+--------+---------------------------+
| /testyssmbak/deeper/2 | UPDATED | String | 2024-06-15 16:38:24+00:00 |
+-----------------------+---------+--------+---------------------------+

And restore:

ssmbak restore /testyssmbak/deeper/2 $UPDATED_MARK

+-----------------------+---------+--------+---------------------------+
| Name                  | Value   | Type   | Modified                  |
+-----------------------+---------+--------+---------------------------+
| /testyssmbak/deeper/2 | UPDATED | String | 2024-06-15 16:38:24+00:00 |
+-----------------------+---------+--------+---------------------------+

Voila. Just /testyssmbak/deeper/2 is UPDATED.

aws ssm get-parameters-by-path --path /testyssmbak --recursive \
  | perl -ne '@h=split; print "$h[4] \t\t $h[6]\n";'

/testyssmbak/1 		 initial
/testyssmbak/2 		 initial
/testyssmbak/3 		 initial
/testyssmbak/deeper/1 		 initial
/testyssmbak/deeper/2 		 UPDATED
/testyssmbak/deeper/3 		 initial

Let's mark the time and clean up our SSM tree:

END_MARK=`date -u +"%Y-%m-%dT%H:%M:%S"`
aws ssm get-parameters-by-path --path /testyssmbak --recursive \
  | perl -ne '@h=split; print "$h[4] ";' \
  | xargs aws ssm delete-parameters --names
sleep 120

DELETEDPARAMETERS       /testyssmbak
DELETEDPARAMETERS       /testyssmbak/1
DELETEDPARAMETERS       /testyssmbak/2
DELETEDPARAMETERS       /testyssmbak/3
DELETEDPARAMETERS       /testyssmbak/deeper/1
DELETEDPARAMETERS       /testyssmbak/deeper/2
DELETEDPARAMETERS       /testyssmbak/deeper/3

And pretend we made a mistake. Oh no! We want them all back. Let's give ssmbak some time to process and see what we can restore.

sleep 120
ssmbak preview /testyssmbak/ $END_MARK --recursive

+-----------------------+---------+--------+---------------------------+
| Name                  | Value   | Type   | Modified                  |
+-----------------------+---------+--------+---------------------------+
| /testyssmbak/1        | initial | String | 2024-06-15 17:34:37+00:00 |
| /testyssmbak/2        | initial | String | 2024-06-15 17:34:37+00:00 |
| /testyssmbak/3        | initial | String | 2024-06-15 17:34:37+00:00 |
| /testyssmbak/deeper/1 | initial | String | 2024-06-15 17:34:37+00:00 |
| /testyssmbak/deeper/2 | UPDATED | String | 2024-06-15 17:35:27+00:00 |
| /testyssmbak/deeper/3 | initial | String | 2024-06-15 17:34:37+00:00 |
+-----------------------+---------+--------+---------------------------+

We won't do the restore after all and stay cleaned-up.

In all this we haven't seen or touched the key /testyssmbak, which differs from path /testyssmbak/.

ssmbak preview /testyssmbak `date -u +"%Y-%m-%dT%H:%M:%S"`

+--------------+---------+--------+---------------------------+
| Name         | Value   | Type   | Modified                  |
+--------------+---------+--------+---------------------------+
| /testyssmbak | initial | String | 2024-06-15 20:55:47+00:00 |
+--------------+---------+--------+---------------------------+

versus:

ssmbak preview /testyssmbak/ `date -u +"%Y-%m-%dT%H:%M:%S"`

+----------------+---------+--------+---------------------------+
| Name           | Value   | Type   | Modified                  |
+----------------+---------+--------+---------------------------+
| /testyssmbak/1 | initial | String | 2024-06-15 21:01:55+00:00 |
| /testyssmbak/2 | initial | String | 2024-06-15 21:01:55+00:00 |
| /testyssmbak/3 | initial | String | 2024-06-15 21:01:55+00:00 |
+----------------+---------+--------+---------------------------+

CLI Gotchas:

• You need a bunch of shady permissions to create the stack. Look for such errors if it fails.

Backup Guarantees

Event Time Preservation

• Regular backups (Create/Update): Event time is preserved via S3 object tags (ssmbakTime), ensuring accurate point-in-time restore even during Lambda processing delays or outages.

• Delete markers: Event time cannot be preserved because S3 delete markers don't support tags. Delete markers use S3's LastModified timestamp (when the Lambda processed the delete) instead of the original event time.

Implications During Outages

If SQS messages queue up during an outage and delete events are processed late:

• Worst case: A parameter that was deleted may appear with its last value instead of showing as deleted when querying for a time between the actual deletion and when the Lambda processed it.

• Safe failure mode: You might restore previously deleted data (resurrection), but you will never lose data that actually existed at the query time.

Example:

• T1: Parameter has value "important"
• T2: Parameter deleted
• T3-T10: Lambda outage (delete event queued)
• T11: Lambda processes delete, creates delete marker with LastModified=T11
• Query at T5: Returns "important" (last backup before T5) instead of showing deleted

This is an inherent limitation of S3 delete markers not supporting tags.

Scripts

• ssmbak-all will back up all SSM params to the bucket. You can also give it a path.

• ssmbak-stack can create, update and give you info about the stack, including all its resources.

• -h for more info.

Seed backups for all previously set SSM Params with ssmbak-all. It will just show you what would be backed-up. --do-it to actually perform the backups.

If you download a new version, best to get that same version running in the Lambda with:

ssmbak-stack <SSMBAK_STACKNAME> update

The lambda is configured to write logs to cloudwatch.

SSMBAK_LAMBDANAME=`ssmbak-stack $SSMBAK_STACKNAME lambdaname`
aws logs tail --format short /aws/lambda/$SSMBAK_LAMBDANAME

2024-06-13T20:11:07 INIT_START Runtime Version: python:3.10.v36	Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:bbd47e5ef4020932b9374e2ab9f9ed3bac502f27e17a031c35d9fb8935cf1f8c
2024-06-13T20:11:07 START RequestId: d404f4c7-1c53-5e41-a7db-aa2248dee8cd Version: $LATEST
2024-06-13T20:11:10 [INFO]	2024-06-13T20:11:10.776Z	d404f4c7-1c53-5e41-a7db-aa2248dee8cd	put_object {'Bucket': 'ssmbak-bucket-vhvs73zpfvy5', 'Key': '/testyssmbak/3', 'Tagging': 'ssmbakTime=1718309456&ssmbakType=String', 'Body': 'initial'}
2024-06-13T20:11:10 [INFO]	2024-06-13T20:11:10.964Z	d404f4c7-1c53-5e41-a7db-aa2248dee8cd	result: 200
2024-06-13T20:11:11 END RequestId: d404f4c7-1c53-5e41-a7db-aa2248dee8cd
2024-06-13T20:11:11 REPORT RequestId: d404f4c7-1c53-5e41-a7db-aa2248dee8cd	Duration: 3430.49 ms	Billed Duration: 3431 ms	Memory Size: 128 MB	Max Memory Used: 84 MB	Init Duration: 282.28 ms
...

Lib Tutorial

Use the cli to get the bucketname, or check the stack resources with your preferred method.

ssmbak-stack ssmbak bucketname

ssmbak-bucket-dkvp9oegrx2y

Session:

>>> from ssmbak.restore.actions import ParamPath
>>> from datetime import datetime, timezone
>>> in_between = datetime.strptime("2024-06-13T01:55:26", "%Y-%m-%dT%H:%M:%S").replace(tzinfo=timezone.utc)
>>> path = ParamPath("/testoossmbak", in_between, "us-west-2", "ssmbak-bucket-dkvp9oegrx2y", recurse=True)
>>> path.preview()
[{'Name': '/testoossmbak/deep/yay', 'Deleted': True, 'Modified': datetime.datetime(2024, 6, 13, 1, 50, 22, tzinfo=tzutc())}]
>>> path.restore()

Development

This is a poetry project, so it should be butter once you get that sorted. Install pre-commit for black on commit, lint and typing on push.

Testing

Testing uses localstack, as you can see in the Github actions. docker-compose up should do the trick, then ./tests/test_localstack.sh.

• source tests/localstack_env.sh to point ssmbak to localstack.

• Recent docker versions allow for docker-compose up --watch, allowing for hot-reloading of the lambda.

• Lambda tests use both the lambda's backup function and hitting the local container running it. Container tests are skipped in AWS.

Testing Gotchas

• When testing on aws instead of localstack, don't use same bucket as running lambda!
  • The lambda will be processing and backing up in addition to the tests.
  • Tests will set versioning on the bucket and manipulate/destroy pytest.test_path.

Addenda

• ssmbak-stack creates two alarms for the process queue, in case you'd like to configure some actions.
• Use a custom kms key for added security, which will require you to set up the infra.
• Support for advanced ssm params has not been tested at all.