Skip to main content

declarative data migration and transformation package

Project description

Data-migrator (version 0.6.2) is a simple data-migration package for python lovers.

https://img.shields.io/pypi/pyversions/data-migrator.svg https://circleci.com/gh/schubergphilis/data-migrator.svg?style=svg
Updates Documentation Status https://api.codacy.com/project/badge/Grade/bf6030e9e7e248979607802880336611 https://api.codacy.com/project/badge/Coverage/bf6030e9e7e248979607802880336611 https://badge.fury.io/py/data-migrator.svg

Data-migrator is a declarative DSL for table driven data transformations, set up as an open and extensive system. Use this to create data transformations for changing databases as a result of changing code, initial loads to datalakes (it contains a Kinesis provider)and more.

Example

Data-migrator assumes data is extracted and loaded with client access.

$ mysql source_db -E 'select id,a,b from table' -B  | python my_filter.py | mysql target_db

It than offers a wide range of primitives with default settings to build complex transformations fast, readable and extendable

from data_migrator import models, transform

class Result(models.Model):
  id   = models.IntField(pos=0) # keep id
  uuid = models.UUIDField()     # generate new uuid4 field
  a    = models.StringField(pos=1, default='NO_NULL', max_length=5, nullable='NULL', replacement=lambda x:x.upper())
  b    = models.StringField(pos=2, name='my_b')

if __name__ == "__main__":
  transform.Transformer(models=[Result]).process()

Installation

Execute the following command to install data-migrator with pip:

pip install data-migrator

See the Installation Instructions in Documentation for more instructions on installing, upgrading, and uninstalling data-migrator.

The project is maintained at GitHub.

Support and contribute

Questions, comments, bug reports and especially tested patches may be submitted directly to the issue tracker.

Everyone interacting with this codebase, issue trackers, chat rooms, and mailing lists is expected to follow the Code of Conduct.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

data-migrator-0.6.2.tar.gz (50.3 kB view details)

Uploaded Source

Built Distribution

data_migrator-0.6.2-py2.py3-none-any.whl (37.2 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file data-migrator-0.6.2.tar.gz.

File metadata

File hashes

Hashes for data-migrator-0.6.2.tar.gz
Algorithm Hash digest
SHA256 f1e7f908aaf70c509b0b567281f861047c92c41606e560c09280c1fc817fee11
MD5 33aa50bbb941125f56a7861f3a4d7b6d
BLAKE2b-256 0d4fec203c44e6f4b0ffcf829961893156c5ae086e6ed1444fd4780384a5a53a

See more details on using hashes here.

File details

Details for the file data_migrator-0.6.2-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for data_migrator-0.6.2-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 51063b3ae01e837cf3ee5adbfbf73fb1409cf74f0a97c75ed50de1af8b9cf639
MD5 8f78836b855fb8e40d05395aa94d2ef0
BLAKE2b-256 66f4b8a3a568e2b522324537ea8658890120756b380bad35b6d2c1e3b77ea1a1

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page