Skip to main content

Migration library for Clickhouse

Project description

Clickhouse Migrator

Clickhouse is known for its scale to store and fetch large datasets.

Development and Maintenance of large-scale db systems many times requires constant changes to the actual DB system. Holding off the scripts to migrate these will be painful.

We found there is nothing existing earlier and developed one inspired by, Flyway, Alembic

This is a python library, which you can execute as a pre-hook using sys python. Or as a migration framework before deployment/server-startup in your application as required.

Publishing to pypi

  • python -m build
  • python -m twine upload --verbose --repository pypi dist/*

Installation

You can install from pypi using pip install clickhouse-migrator.

Usage

# <= v1.0.4
from migration_lib.migrate import migrate

migrate(db_name, migrations_home, db_host, db_user, db_password, create_db_if_no_exists)
# > v1.0.4
from clickhouse_migrate.migrate import migrate

migrate(db_name, migrations_home, db_host, db_user, db_password, create_db_if_no_exists)
Parameter Description Default
db_name Clickhouse database name None
migrations_home Path to list of migration files <project_root>
db_host Clickhouse database hostname localhost
db_password ***** ****
create_db_if_no_exists If the db_name is not present, enabling this will create the db True
db_port Database port incase your server runs on a non default port None (defaults to 9000)
queue_exec Command Pipelining (wait for system mutations if any to complete) for every command True

Folder and Migration file patterns

The filenames are pretty similar to how flyway keeps it.

Your first version filename should be prefixed with V1__ (double underscore) These migrations are executed one by one, failures in between will stop and not further version files will be executed.

Multi statement and single statement migrations

If your migration is a single statement, you can create a file in the migration folder using the .sql extension and push your migration statement in there.

If you want to execute more than one statement in your migration, you can use a json file using the array syntax. Note that when using a json file, contents should be a valid json array as show. Ensure to keep migrations logical. Its not a good practise to push all migrations to one json file and neither is it wise to in all cases have them each statement in one file.

[
  "CREATE TABLE pytest.sample1(id UInt32, name String) ENGINE MergeTree PARTITION BY tuple() ORDER BY tuple()",
  "CREATE TABLE pytest.sample2(id UInt32, name String) ENGINE MergeTree PARTITION BY tuple() ORDER BY tuple()",
  "CREATE TABLE pytest.sample3(id UInt32, name String) ENGINE MergeTree PARTITION BY tuple() ORDER BY tuple()"
]

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

clickhouse-migrator-1.0.6.tar.gz (4.7 kB view details)

Uploaded Source

Built Distribution

clickhouse_migrator-1.0.6-py3-none-any.whl (5.4 kB view details)

Uploaded Python 3

File details

Details for the file clickhouse-migrator-1.0.6.tar.gz.

File metadata

  • Download URL: clickhouse-migrator-1.0.6.tar.gz
  • Upload date:
  • Size: 4.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.3 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.6.8

File hashes

Hashes for clickhouse-migrator-1.0.6.tar.gz
Algorithm Hash digest
SHA256 e23f4e975774ffbc566ffcdc6350d675fe32910560ee0fb06b6009d040258aa0
MD5 04da898911830da34ce76bbcd70f2fb0
BLAKE2b-256 7385b4b4e13f7338f670d033a4cc52e17282dafcdb3c03994a70e293378f65ec

See more details on using hashes here.

File details

Details for the file clickhouse_migrator-1.0.6-py3-none-any.whl.

File metadata

  • Download URL: clickhouse_migrator-1.0.6-py3-none-any.whl
  • Upload date:
  • Size: 5.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.7.1 importlib_metadata/4.8.3 pkginfo/1.8.2 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.6.8

File hashes

Hashes for clickhouse_migrator-1.0.6-py3-none-any.whl
Algorithm Hash digest
SHA256 ebb77222597168307ea38643f58d8d5d3165878818342b15fb1603a4a4809f6d
MD5 925453c2877bd535b81210599d8ae51c
BLAKE2b-256 85abbfe093c865352846c04749e1d64acebcffbbd5804497a42cede87ae7e4a0

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page