Skip to main content
https://travis-ci.org/cocreators-ee/migrate-anything.svg?branch=master https://img.shields.io/badge/code%20style-black-000000.svg https://codecov.io/gh/cocreators-ee/migrate-anything/branch/master/graph/badge.svg https://sonarcloud.io/api/project_badges/measure?project=cocreators_migrate-anything&metric=alert_status GitHub issues PyPI - Downloads PyPI PyPI - Python Version https://img.shields.io/badge/License-BSD%203--Clause-blue.svg

Migrate anything - database (etc.) migration utility, especially for Python projects.

What is this?

It’s kinda annoying how often you run into the question of how to handle migrations in your project, and there hasn’t seem to emerged any good, DB -agnostic, framework-agnostic, and storage-agnostic tool to manage them.

This project is an attempt to change that.

Basically what it does when you run migrate-anything migrations is:

  1. Find all the files migrations/*.py and sort them

  2. Any that are not yet registered in the DB will be loaded, their up() is executed, and the file’s contents stored in the DB

  3. Any files that are missing from the fs but are in the DB will have their code loaded from the DB and their down() is executed - in reverse order

License

Licensing is important. This project uses BSD 3-clause license, and adds no other dependencies to your project (it does use a few things during build & testing) - that’s about as simple, safe, and free to use as it gets.

For more information check the LICENSE -file.

Usage examples

Basic usage

Firstly you’ll need this package in your project. Pick one of these:

pip install -U migrate-anything
poetry add migrate-anything
pipenv install migrate-anything

Simply put, create a Python package, don’t be too clever and call it e.g. migrations. Then put files in that package:

# migrations/__init__.py
from migrate_anything import configure, CSVStorage

configure(storage=CSVStorage("migration_status.csv"))
# migrations/01-initialize-db.py
# Please note that this is built for a completely hypothetical DB layer
from my_db import get_db

DB = get_db()

def up():
    DB.create_table("example")

def down():
    DB.delete_table("example")

This would configure your migrations’ status to be stored in a local file called migration_status.csv and set up your first migration script. If you have a my_db module that works like this you could just run this with a single command:

migrate-anything migrations
poetry run migrate-anything migrations
pipenv run migrate-anything migrations

Now in the real world you might want something more durable and a realistic example, so here’s e.g. what you’d do when using MongoDB:

# __init__.py
from migrate_anything import configure, MongoDBStorage
import pymongo

db = pymongo.MongoClient().my_db

configure(storage=MongoDBStorage(db.migrations))
# 01-initialize-db.py
from pymongo import MongoClient

client = MongoClient()
db = client.my_db

def up():
    db.posts.insert_one({
        "id": "post-1",
        "title": "We're live!",
        "content": "This is our first post, yay."
    })
    db.posts.create_index("id")

def down():
    db.posts.drop()

This would configure storage to a my_db.migrations MongoDB collection.

Command line flags

# Revert the last migration using migration code file.
migrate-anything migrations --revert-latest

Custom storage engines

Writing your own custom storage engine is easy.

# __init__.py
from migrate_anything import configure


class CustomStorage(object):
    def __init__(self, file):
        self.file = file

    def save_migration(self, name, code):
        with open(self.file, "a", encoding="utf-8") as file:
            file.write("{},{}\n".format(name, code))

    def list_migrations(self):
        try:
            with open(self.file, encoding="utf-8") as file:
                return [
                    line.split(",")
                    for line in file.readlines()
                    if line.strip()  # Skip empty lines
                ]
        except FileNotFoundError:
            return []

    def remove_migration(self, name):
        migrations = [
            migration for migration in self.list_migrations() if migration[0] != name
        ]

        with open(self.file, "w", encoding="utf-8") as file:
            for row in migrations:
                file.write("{},{}\n".format(*row))


configure(storage=CustomStorage("test.txt"))

You can also check out the examples.

Contributing

This project is run on GitHub using the issue tracking and pull requests here. If you want to contribute, feel free to submit issues (incl. feature requests) or PRs here.

You will need pre-commit set up to make contributions.

To set up development tools for this, run:

pre-commit install
virtualenv .venv

.venv/bin/activate
# OR
.venv\Scripts\activate.bat

pip install -r dev-requirements.txt
pip install -e .

And then to run the tests

pytest

When you have improvements to make, commit (and include any cleanup pre-commit might do), push your changes to your own fork, make a PR.

Future ideas

Future ideas include support for other DB engines (feel free to contribute), and Kubernetes ConfigMap. Annoyingly storage to Kubernetes from inside a pod and in code is not quite as simple as just running kubectl.

Oh and your Kubernetes pods will likely require the necessary RBAC rules to manage their ConfigMap. It’s unfortunately kinda complex, but I’m sure you can figure it out e.g. with this guide.

Financial support

This project has been made possible thanks to Cocreators and Lietu. You can help us continue our open source work by supporting us on Buy me a coffee.

https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png

Metadata

Release files for migrate-anything 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for migrate-anything 0.2.0
File Size Uploaded Provenance Actions
migrate_anything-0.2.0.tar.gz 11.6 kB Without provenance Details for migrate_anything-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for migrate-anything 0.2.0
File Interpreter ABI Platform Actions
migrate_anything-0.2.0-py2.py3-none-any.whl Python 3, Python 2 none any Details for migrate_anything-0.2.0-py2.py3-none-any.whl

Total release size: 21.8 kB

Release files / migrate_anything-0.2.0.tar.gz

Download URL migrate_anything-0.2.0.tar.gz
Size 11.6 kB
Tags Source
SHA-256 checksum
How to use checksums
8072b3e3e406294368dcf68db5ffcfdbe238b0efe4a6aa17d9de2f48d91e9959
BLAKE2b-256 checksum
How to use checksums
9f9daa0a8db8ab8c82be652fe1bcf95f4b94f0c9376e7d1dc04dced482c5ff56
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.7

Release files / migrate_anything-0.2.0-py2.py3-none-any.whl

Download URL migrate_anything-0.2.0-py2.py3-none-any.whl
Size 10.1 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums
6b28e49533116a3f4c210d227f60bdf205b19b15145e81116faf1eb489b76264
BLAKE2b-256 checksum
How to use checksums
31915daaaff4f245731faf8fc3d027a67a44ebd24222beeec036522adc0049be
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.7

Security for migrate-anything 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page