Skip to main content

Indexima migration schema based on yoyo

Project description


Unix Build Status PyPI Version PyPI License

Versions following Semantic Versioning


Indexima migration schema based on yoyo and pyhive.

The little story

In the land of database migration tool, i have tried:

  • flyway
  • liquidbase with hive extention

Both either did not support hive (as flyway), or indexima did not fully compliant with hive (wich cause probleme with liquidbase)

So I try to found a module not too complex in order to migrate our indexima schema in a safe way.

In this early release, I just trying to do the job.



  • Python 3.7+


Install this library directly into an activated virtual environment:

$ pip install yoyo-indexima

or add it to your Poetry project:

$ poetry add yoyo-indexima


Hive connection

  1. backend ui must start with indexima://
  2. If you have trouble to obtain an hive connection, please read

Note: If you using python in docker, you should install :

apt-get update -qq
apt-get install -qqy gcc libsasl2-dev libsasl2-2 libsasl2-modules-gssapi-mit 


You could see a complete sample under 'example' folder.

using python client

usage: yoyo_indexima [-h] [-s SOURCE] -u URI {show,apply}


yoyo_indexima  apply  -s $(pwd)/example/migrations/ -u "indexima://admin:super_password@localhost:10000/default"


  • show Show migrations
  • apply Apply migrations
  • reapply Reapply migrations
  • rollback Rollback migrations
  • mark Mark migrations as applied, without running them
  • unmark Unmark applied migrations, without rolling them back
  • break-lock Break migration locks

Help for apply:

> yoyo_indexima  apply -h
usage: yoyo_indexima apply [-h] [-s SOURCE] -u URI [-f] [-a] [-r REVISION]

optional arguments:
  -h, --help            show this help message and exit
  -s SOURCE, --source SOURCE
                        source path of migration script (default ./migrations)
  -u URI, --uri URI     backend uri
  -f, --force           Force apply/rollback of steps even if previous steps
                        have failed
  -a, --all             Select all migrations, regardless of whether they have
                        been previously applied
  -r REVISION, --revision REVISION
                        Apply/rollback migration with id REVISION
  -d, --dry-run         Dry run: no modification will be applied

within python code

If your migrations script are under directory migration folder

import os

from yoyo_indexima import get_backend, read_migrations

if __name__ == "__main__":

    # obtain IndeximaBackend
    backend = get_backend('indexima://admin:super_password@localhost:10000/default?auth=CUSTOM')

    # Read migrations folder
    migrations = read_migrations(os.path.join(os.getcwd(), 'migrations/**/*'))
    print(f'migrations: {migrations}')
    if migrations:
        # apply migration
        with backend.lock():

Management table

This tool create in your default schema:

  • a log table: 'yoyo_log'
  • a lock_table: 'yoyo_lock'
  • a version table: 'yoyo_version'
  • a migration table: 'yoyo_migration'

Migration script template


from yoyo import step

__depends__ = {{{depends}}}

steps = [
    step("create ...", "drop ...")

Configure hive connection

In python script, on IndeximaBackendinstance, you could use:

  • set_hive_configuration: A dictionary of Hive settings (functionally same as the set command)
  • set_hive_thrift_transport: an instance of TSaslClientTransport

As see in, you could do things like that:

import sasl
from thrift_sasl import TSaslClientTransport
from thrift.transport.TSocket import TSocket

def create_hive_plain_transport(host, port, username, password, timeout=60):
    socket = TSocket(host, port)
    socket.setTimeout(timeout * 1000)

    sasl_auth = 'PLAIN'

    def sasl_factory():
        sasl_client = sasl.Client()
        sasl_client.setAttr('host', host)
        sasl_client.setAttr('username', username)
        sasl_client.setAttr('password', password)
        return sasl_client

    return TSaslClientTransport(sasl_factory, sasl_auth, socket)

backend = get_backend('indexima://admin:super_password@localhost:10000/default?auth=CUSTOM')

Extends IndeximaBackend

If you extends IndeximaBackend you could register your classes, in the function: get_backend(uri: str, backend=IndeximaBackend) -> DatabaseBackend:

TODO: add a client parameter to specify full class name in cli.


The MIT License (MIT)


See Contributing

Next step

  • production usage in order to see how this tool made the job
  • more documentation in code
  • purpose few change in 'yoyo' like set all SQL command on Backend class
  • ...

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

yoyo_indexima-0.1.3.tar.gz (15.2 kB view hashes)

Uploaded Source

Built Distribution

yoyo_indexima-0.1.3-py3-none-any.whl (16.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page