L3-level cdk constructs for DMS
Project description
DMS Patterns - a library to facilitate migrations
This library aims at simplifying the task of setting up a migration project using AWS DMS and the AWS CDK, by providing L2 constructs that take care of creating the necessary roles and resources and high-level L3 constructs that represent migration patterns from a database (in the cloud or a DB in AWS RDS) to AWS s3. See the example section below for more details.
This library is written using the wonderful projen framework.
Note: this library is just the result of some personal experimentation. It is not an official AWS library and is not supported by AWS!
Installation
The library is available on npmjs.com and can be installed using:
npm i dms-patterns
And on pypi:
pip install dms-patterns
Usage Examples
Migrating data from MySQL to S3
This section demonstrates creating a stack that migrates data from a MySQL database to S3. The stack is created in TypeScript, but the same constructs are available in Python. We start by adding a source endpoint to our stack:
this.source = new MySqlEndpoint(this, 'SourceEndpoint', {
endpointType: EndpointType.SOURCE,
databaseName: *******,
endpointIdentifier: 'mysqlEndpoint',
mySqlSourceEndpointSettings: {
secretsManagerSecretId: 'arn:aws:secretsmanager:**********',
},
});
A MySqlEndpoint is just a dms.CfnEndpoint, but the construct also takes care of initializing a secretsManagerAccessRole used by DMS to read the specified secret, that contains the necessary connection details:
{
"password":"*****",
"username":"*****",
"port":3306,
"host":"****"
}
Our target is :
this.target = new S3TargetEndpoint(this, 'S3Target', {
bucketArn: props.bucketArn,
});
the construct takes again care of creating the necessary role for creating objects within the given bucket.
We can now define the replication itself. In case no default (subnet) vpc exists in your account, you may want to create a replicationSubnetGroup that describes where your resources will be deployed:
const replicationSubnetGroup = new dms.CfnReplicationSubnetGroup(this, 'ReplicationSubnetGroup', {
replicationSubnetGroupDescription: 'ReplicationSubnetGroup',
replicationSubnetGroupIdentifier: 'mysqlreplicationsubnetID',
subnetIds: [
'subnet-******',
'subnet-******',
'subnet-******',
],
});
and, finally, a computeConfig:
const computeConfig: dms.CfnReplicationConfig.ComputeConfigProperty = {
minCapacityUnits: CapacityUnits._1,
maxCapacityUnits: CapacityUnits._2,
multiAz: false,
replicationSubnetGroupId: replicationSubnetGroup.replicationSubnetGroupIdentifier,
vpcSecurityGroupIds: [
replicationSecurityGroup.securityGroupId,
],
};
and some rules for e.g. selecting the right tables:
const tableMappings = new TableMappings(
[
new SelectionRule(
{
objectLocator: {
schemaName: 'schemaname',
tableName: 'tableName',
},
ruleAction: SelectionAction.INCLUDE,
},
),
],
);
The tableMappings object takes care of assigning the rules a name and id if not specified and comes with a format method, see below. Finally, we can write the replication in itself:
new dms.CfnReplicationConfig(this, 'ReplicationConfig', {
computeConfig: computeConfig,
replicationConfigIdentifier: 'replicationConfigIdentifier',
replicationType: ReplicationTypes.FULL_LOAD,
sourceEndpointArn: this.source.ref,
tableMappings: tableMappings.format(),
targetEndpointArn: this.target.ref,
});
In the example above the 'experiment' table of the database specified above will be migrated to S3, and after executing the migration, a file 'LOAD00000001' is created in the specified 'folder' in the S3 bucket.
So far I demonstrated a couple of L2 constructs for the endpoints; but an L3 construct is also available that in turn takes care of creating all the endpoints and the migration itself:
const mySql2S3 = new MySql2S3(this, 'mysql2S3', {
databaseName: ******,
mySqlSourceEndpointSettings: {
secretsManagerSecretId: 'arn:aws:secretsmanager:*******',
},
bucketArn: bucketArn,
tableMappings: tableMappings,
computeConfig: computeConfig,
replicationConfigIdentifier: '*****',
});
which makes migrations easier. A similar pattern exists for postgres, and more could be added in future versions.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dms_patterns-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 10615937d7e0c0374bc557e31dce7e18662bb03d64597b29a53f8b7fb86f72ca |
|
MD5 | 92fcf0678dee8fa4d1ba402ce2802aca |
|
BLAKE2b-256 | 36ab25d8bb7a13f6973023de24bbdd7950535c9c72fa1e7f7e3a8dcbf28df2e1 |