django-migrations-ci 0.0.4

Django migrations CI optimization

django-migrations-ci

Optimizations to run less migrations on CI.

Django migrations are slow because of state recreation for every migration and other internal Django magic.

In the past, I tried to optimize that on Django core, but learnt it's a running issue.

Assumptions

1. I want to run my migrations on CI. It is a sanity check, even if they are generated by Django in some cases.
2. I don't have to run migrations all the time. If migrations didn't change, I can reuse them.
3. The solutions probably are CI-specific and database-specific, but it is possible to write code for a generic workflow and snippets to support each CI.

Install

Install the package with pip:

pip install django-migrations-ci

Add django_migrations_ci to Django settings INSTALLED_APPS.

INSTALLED_APPS = [
    ...,  # other packages
    "django_migrations_ci",
]

How to use

The command migrateci execute all migrations and generate dump files migrateci-* to be cached on CI.

If these files already exist on disk, they are used to prepare the database without running all migrations again.

Configure your CI to cache these migrateci-* files, based on migration files. /

Workflow

This is how the "run test" CI job should work.

Restore migrateci.sql from CI cache

if migrateci.sql exists on cache:
  Restore migrateci.sql to test database
else:
  Setup test database
  Dump test database to migrateci.sql

Clone the test database to run threaded tests

Save migrateci.sql to CI cache

Cache example on GitHub

    steps:
    - uses: actions/cache@v3
      name: Cache migrations
      with:
        path: migrateci-*
        key: ${{ secrets.EXAMPLE_CACHE_PREFIX }}-${{ hashFiles('**/migrations/*.py') }}
    - name: Migrate database
      run: ./manage.py migrateci --parallel $(nproc)
    - name: Test with Django
      run: ./manage.py test --keepdb --parallel $(nproc)

Cache example on GitLab

Still have to abstract psql/pg_dump/pg_restore, but I expect something like that will work:

test_job:
  stage: test
  script:
    - ./manage.py migrateci $(nproc)
    - ./manage.py test --keepdb --parallel $(nproc)
  cache:
    key:
      # GitLab docs say it accepts only two files, but for some reason it works with wildcards too.
      # You can't add more than two lines here.
      files:
        - "requirements.txt"
        - "*/migrations/*.py"
    paths:
      - migrateci-*

How databases for parallel tests are named

Django test framework has a --parallel N flag to test with N parallel processes, naming databases from 1 to N.

• On sqlite3, a db.sqlite3 generate db_N.sqlite3 files.
• On PostgreSQL, a db generate test_db_N.

Pytest pytest-django use pytest-xdist for parallel support, naming databases from 0 to N-1.

• On sqlite3, a db.sqlite3 generate db.sqlite3_gwN files.
• On PostgreSQL, a db generate test_db_gwN.