madmigration 0.1.8

Database Migration Tool

Don't use this tool in production databases!! At this time tool under development!!

🧐 About

The Database Migration Tool was designed for those looking to migrate their data from one database to another. Basically, the tool is focused on transferring data from different database structures. Currently, the MySQL, Mariadb and Postgres driver related tool allows us to add NoSQL to SQL databases and vice versa. Our main goal is to make data migration possible in all environments.

Documentation: Documentation

🏁 Getting Started

Installing

pip install madmigration

🎈 Usage

After installation you should define YAML file where configuration will be taken in order to apply data to target database. Yaml file with list and dictionaries may contain following structures:

Connection Config

• SourceConfig is intented to be data from source database
• DestinationConfig is intented to be transfered data to target database

version: 0.1.6
Configs:
  - SourceConfig:
       dbURI: "postgres://root:admin@127.0.0.1/oldDB"
  - DestinationConfig:
       dbURI: "mysql://root:admin@127.0.0.1/newDB"

migrationTables:
  - migrationTable:
      SourceTable:
        name: users
      DestinationTable:
        name: persons
        create: True

      MigrationColumns:
        - sourceColumn:
            name: id
          destinationColumn:
            name: id
            options:
              type_cast: bigint
              primary_key: true
              autoincrement: true

        - sourceColumn:
            name: name
          destinationColumn:
            name: fname
            options:
              type_cast: varchar
              length: 32

        - sourceColumn:
            name: surname
          destinationColumn:
            name: lname
            options:
              type_cast: varchar
              length: 32
              index: true

        - sourceColumn:
            name: age
          destinationColumn:
            name: age
            options:
              type_cast: int

        - sourceColumn:
            name: createdAT
          destinationColumn:
            name: created_at
            options:
              type_cast: datetime

        - sourceColumn:
            name: updatedAT
          destinationColumn:
            name: updated_at
            options:
              type_cast: datetime

Configs section

• SourceConfig set the source database database configurations
  • dbURI source database URI
• DestinationConfig set the destination database configurations
  • dbURI destination database URI

Configs:
  - SourceConfig:
       dbURI: "postgres://root:admin@127.0.0.1/oldDB"  # set source database uri
  - DestinationConfig:
       dbURI: "mysql://root:admin@127.0.0.1/newDB"  # set destination database uri

migrationTables section

• migrationTables in this configuration, you will write the source of the table that you have to migrate and the destination tables that will migrate the data.
  • migrationTable specify the source and destination table name
    • SourceTable information about source table
      • name source table name
    • DestinationTable information about destination table
      • name destination table name
      • create bool value. This parameter tells the program whether it should create a table or not. (default false)

migrationTables:
  - migrationTable:
      SourceTable:
        name: users
      DestinationTable:
        name: persons
        create: True

MigrationColumns section

• MigrationColumns specify source and destination column
  • sourceColumn information about source column
    • name source column name
  • destinationColumn information about destination column
    • name destination column name
    • options column options
      • type_cast destination column type name varchar,integer etc. (when we convert data we use this parameter)

MigrationColumns:
  - sourceColumn:
      name: id
    destinationColumn:
      name: id
      options:
        type_cast: bigint
        primary_key: true
        autoincrement: true

If you want to create a foreign key you can specify it in the column parameters

- sourceColumn:
    name: USERID
  destinationColumn:
    name: user_id
    options:
      type_cast: uuid
      foreign_key:
        table_name: users
        column_name: id
        ondelete: CASCADE

You can split your .yaml files or import .json file into .yaml file.

You must create the main .yaml file and importing other files into main .yaml file.

main.yaml file

version: 1.1
Configs:
  - SourceConfig:
      dbURI: "mysql://root:admin@127.0.0.1/old"
  - DestinationConfig:
      dbURI: "postgresql://root:admin@127.0.0.1/new"

migrationTables:
  - migrationTable: !import company.yaml
  - migrationTable: !import op_cond.json

company.yaml file

SourceTable:
  name: company
DestinationTable:
  name: company
  create: true 

MigrationColumns:
  - sourceColumn:
      name: id
    destinationColumn: 
      name: id
      options:
        primary_key: true
        type_cast: uuid

  - sourceColumn:
      name: name
    destinationColumn:
      name: name
      options:
        length: 120
        type_cast: varchar
        nullable: false

  - sourceColumn:
      name: created
    destinationColumn:
      name: created
      options:
        type_cast: datetime
  - sourceColumn:
      name: updated
    destinationColumn:
      name: updated
      options:
        type_cast: datetime
        nullable: true

op_conds.json file

{
    "SourceTable": {
      "name": "operation_conditions"
    },
    "DestinationTable": {
      "name": "operation_conditions",
      "create": true
    },
    "MigrationColumns": [
      {
        "sourceColumn": {
          "name": "id"
        },
        "destinationColumn": {
          "name": "id",
          "options": {
            "primary_key": true,
            "type_cast": "uuid"
          }
        }
      },
      {
        "sourceColumn": {
          "name": "interest"
        },
        "destinationColumn": {
          "name": "interest",
          "options": {
            "type_cast": "varchar",
            "length": 30,
            "nullable": false
          }
        }
      },
      {
        "sourceColumn": {
          "name": "FIFD"
        },
        "destinationColumn": {
          "name": "FIFD",
          "options": {
            "type_cast": "varchar",
            "length": 30,
            "nullable": false
          }
        }
      },
      {
        "sourceColumn": {
          "name": "comission"
        },
        "destinationColumn": {
          "name": "comission",
          "options": {
            "type_cast": "varchar",
            "length": 30,
            "nullable": false
          }
        }
      }
    ]
  }

Currently new feature PostgreSql to MongoDB has been added.

version: 1.1
Configs:
  - SourceConfig:
      dbURI: "postgresql://sabuhi:sabuhi12345@localhost:5432/company_service"
  - DestinationConfig:
      dbURI: "mongodb://localhost:27017/mydb"

migrationTables:
  - migrationTable:
      SourceTable:  #postgresql table name
        name: company  
      DestinationTable: #the collection name:
        name: Company

      MigrationColumns:
        - sourceColumn:
            name: id
          destinationColumn:
            name: id
            options:
              type_cast: uuid

        - sourceColumn:
            name: name
          destinationColumn:
            name: NAME
            options:
              type_cast: varchar

        - sourceColumn:
            name: created
          destinationColumn:
            name: CREATED
            options:    
              type_cast: datetime
        - sourceColumn:
            name: email
          destinationColumn:
            name: EMAIL
            options:     
              type_cast: string
        - sourceColumn:
            name: updated
          destinationColumn:
            name: UPDATED
            options:
              type_cast: datetime
         - sourceColumn:
            name: code
          destinationColumn:
            name: CODE
            options:      
              type_cast: string

Work on PostgreSQl to MongoDB still goes, we will add other futures as soon as possible.

We will create all tables and database on the destination server if they do not exist

madmigrate -f migration_schema.yaml

✍️ Authors

• @AliyevH - Idea & Initial work
• @Turall
• @sabuhish

See also the list of contributors who participated in this project.

Contributing

We are open to new ideas, additions. If you have any we would be happy to recieve and diccuss.

• Contributing