hyperscale-crypto

Application-tier encrypted Django fields backed by Tink keysets and AWS KMS.

Install the hyperscale-crypto package; import it as hyperscale.crypto.

Values are encrypted in Python before they reach the database. Two Tink keysets hold the data keys: aead (AES256-GCM, randomised) and daead (AES256-SIV, deterministic). Both are stored in your repository, wrapped by a KMS key, and unwrapped in memory at startup.

Install

uv add hyperscale-crypto

INSTALLED_APPS = [
    # ...
    "hyperscale.crypto",
]

Then run python manage.py migrate. The app ships migrations for three tables: KeysetState (the active deterministic key), KeysetCanary (the startup check) and KeyRotationEvent (the rotation log).

Settings

import os
from pathlib import Path

BASE_DIR = Path(__file__).resolve().parent.parent

HYPERSCALE_CRYPTO = {
    "mode": os.environ.get("CRYPTO_MODE", "kms"),  # "kms" or "cleartext"
    "kms_key_uri": os.environ.get("CRYPTO_KMS_KEY_URI", ""),
    "keyset_dir": BASE_DIR / "keysets",  # required
    "reunwrap_seconds": 900,
    "revoke_on_reunwrap_failure": False,
    "retention_days": 35,
    "git_sha": os.environ.get("GIT_SHA", ""),
}

reunwrap_seconds and retention_days must be positive integers and revoke_on_reunwrap_failure a real bool (not the string "false"). In kms mode, kms_key_uri must start with aws-kms:// (or fake-kms:// for tests). mode="cleartext" and fake-kms:// key URIs are only accepted when the ENV environment variable is development or test. Any misconfiguration (missing or unknown keys, a bad value, cleartext outside those environments) raises ImproperlyConfigured during Django setup, not on first use.

Keysets

python manage.py keyset_init              # both keysets; --keyset aead|daead|all

In kms mode this writes <keyset_dir>/aead.json and <keyset_dir>/daead.json, each wrapped by the KMS key. Commit them: the wrapped files are safe to store, and every deploy ships the keys it needs. Protect the keyset path with branch protection and signed commits, since whoever can change these files can change which keys the application trusts.

In cleartext mode the files are <keyset_dir>/dev/aead.json and <keyset_dir>/dev/daead.json, unwrapped, for development and tests only.

keyset_init refuses to overwrite a keyset that already exists.

Fields

from django.db import models

from hyperscale.crypto import fields as ef


class Customer(models.Model):
    owner_id = models.IntegerField()
    notes = ef.EncryptedTextField(blank=True, default="")
    token = ef.EncryptedCharField(
        max_length=64, context=lambda obj: f"owner:{obj.owner_id}"
    )
    ni_number = ef.DeterministicEncryptedCharField(max_length=9, unique=True)

• Storage. Every field is a text column holding hc1: followed by the URL-safe base64 Tink ciphertext. None and "" are stored as they are. max_length and the other validators apply to the plaintext.
• Reading. Values decrypt lazily on first attribute access. .values() and .values_list() return the stored value wrapped in the hyperscale.crypto.fields.Ciphertext marker (a str subclass), not the plaintext. To copy a stored value unchanged, assign that Ciphertext instance; a plain str is always encrypted.
• Lookups. Randomised fields support isnull only. Deterministic fields support exact, in and isnull; any other lookup or transform raises FieldError. Ordering is unsupported: it would sort by ciphertext. db_index is rejected on every encrypted field, and unique=True on a randomised field (equal values encrypt differently, so the database cannot enforce it); use the deterministic variant. auto_now and auto_now_add are rejected on EncryptedDateField.
• Bulk writes. QuerySet.update(), bulk_update() and raw saves (such as loaddata) encrypt plaintext values too. A field with a context cannot be written that way (there is no instance to compute the context from) and raises FieldError; save the instances instead. Query expressions are refused (FieldError) because their result would be stored unencrypted or would corrupt the ciphertext; the exceptions are F() of the same field, Value(x, output_field=<the field>) and the Case/When tree bulk_update() builds from such values.
• context (randomised fields only) is a callable from the model instance to a string that is bound into the ciphertext. It must be a stable property of the row, such as an owner or tenant id. If a context input must change, read the field before changing it and save both together. Saving a row whose context changed while the field was never read raises DecryptionError (naming the model, field and pk) instead of storing a value that would no longer decrypt. Context fields cannot be read or written from a migration (historical models do not carry the callable); that raises FieldError. Use a management command with the live model.
• Deterministic fields require unique=True and reject context. Every deterministic write (save() of a new or loaded row, QuerySet.update(), bulk_update()) must run inside a transaction, on every backend; outside one it raises TransactionManagementError. On PostgreSQL the write also takes a share lock on the KeysetState row. Set ATOMIC_REQUESTS = True on the database and wrap background writers in transaction.atomic().

Stores

Some secrets live where a field cannot: inside a third-party model's JSON column, for example. An encrypted store encrypts and decrypts such values under the aead keyset, bound to a label, and tells rotation where they are, so keyset_reencrypt, keyset_retire and keyset_status cover them like any field.

from hyperscale.crypto import registry
from hyperscale.crypto.encoding import is_encoded
from hyperscale.crypto.stores import EncryptedStore


class TotpSecrets(EncryptedStore):
    def __init__(self):
        super().__init__("mfa.Authenticator.data")  # app.Model.field

    def rows(self):
        qs = Authenticator._base_manager.order_by("pk").values_list("pk", "data")
        for pk, data in qs.iterator(chunk_size=1000):
            yield pk, [data["secret"]] if is_encoded(data.get("secret")) else []

    def rewrite(self, pk, transform):
        row = Authenticator._base_manager.select_for_update().get(pk=pk)
        new = transform(row.data["secret"])
        if new == row.data["secret"]:
            return False
        row.data["secret"] = new
        row.save(update_fields=["data"])
        return True


TOTP_SECRETS = TotpSecrets()
# in AppConfig.ready():
registry.register_store(TOTP_SECRETS)

• encrypt(text) returns an hc1: string; decrypt(stored) returns the text, or raises DecryptionError for anything that is not this store's ciphertext, cleartext included.
• The label is the associated data: a value copied to another store, column or app fails to decrypt.
• rows() yields (pk, [stored, ...]) for every row that may hold ciphertexts, without decrypting, through the base manager. Name the slots explicitly: never scan a JSON document for hc1: strings, because a value a user controls could then stall a rotation.
• rewrite(pk, transform) runs inside rotation's transaction: lock the row, apply transform to each ciphertext, save if anything changed, return whether it did.
• Stores are randomised only (no deterministic variant); they appear in rows_by_key under aead, and keyset_retire refuses while a store still has rows under the key.

Rotation

Randomised (aead) keys rotate online:

python manage.py keyset_rotate --keyset aead      # new primary key
# commit aead.json and deploy everywhere
python manage.py keyset_reencrypt                 # --batch-size 500 (>0); resumable
python manage.py keyset_retire aead <old_key_id>  # refuses while rows use it
# after retention_days:
python manage.py keyset_destroy aead <old_key_id>

Deterministic (daead) keys switch in one step, because lookups must find every row under a single key:

python manage.py keyset_add_key --keyset daead    # enabled, not active
# commit daead.json and deploy everywhere
python manage.py keyset_rotate --keyset daead --to <new_key_id> [--yes] [--lock-timeout 30]
python manage.py keyset_retire daead <old_key_id>
# after retention_days:
python manage.py keyset_destroy daead <old_key_id>

The daead switch prints the row counts, asks you to type confirm (unless --yes), then re-encrypts every deterministic value in one transaction. It locks the KeysetState row for that transaction (on PostgreSQL, the table in EXCLUSIVE mode first, so the switch queues fairly behind in-flight writers instead of starving), and every deterministic writer waits until it commits. Reads are not blocked. --lock-timeout <seconds> (PostgreSQL) gives up cleanly, changing nothing, if in-flight writers hold the lock longer than that. On large tables, run it in a maintenance window.

keyset_retire refuses while any row is still encrypted under the key, counting the startup canary row as one: after an aead rotation run keyset_reencrypt (it moves the canary too) even if no model has data yet.

keyset_destroy refuses until retention_days have passed since the key was retired. Set retention_days to cover your database backup window: a backup that still holds values under a destroyed key cannot be decrypted.

Every command records a KeyRotationEvent row (init, add_key, rotate, reencrypt, retire, destroy) with the key id, the actor, details such as row counts, and git_sha. Pass --actor <name> to record who ran it; it defaults to the OS user.

Evidence

python manage.py keyset_status          # human-readable
python manage.py keyset_status --json   # machine-readable

The JSON has these top-level keys:

• keysets: per keyset, every key id with its status (ENABLED, DISABLED, DESTROYED) and whether it is primary and (daead) active.
• rows_by_key: per keyset, per field (app.Model.field) and, under aead, per registered store, the number of rows under each key id, read from ciphertext prefixes without decrypting. The canary row is listed too, as hyperscale_crypto.KeysetCanary.aead_value / .daead_value.
• last_events: the time of the most recent event of each action, or null.
• canary_ok, and canary_error when the canary check failed.
• state_matches_file: whether the database's active daead key is an enabled key in the shipped daead.json.

Startup

When the app is ready, it runs a canary check: it decrypts a known value stored in KeysetCanary with both keysets (writing the value on first run) and raises ImproperlyConfigured, naming the keyset, if either keyset cannot be loaded (a missing file, or one that does not unwrap) or cannot decrypt it. This catches a wrong KMS key, a swapped keyset file or a database restored under different keys before any request is served. The check is skipped for makemigrations, migrate, collectstatic, check, showmigrations and sqlmigrate, under pytest, and before the canary table has been migrated.

The check runs on a plain thread, so it also works under servers that import the ASGI module inside a running event loop (uvicorn, gunicorn's UvicornWorker), where Django refuses database calls on the loop thread.

Startup neither arms the re-unwrap timer nor keeps its database connection: under a pre-fork server (gunicorn, uWSGI) it runs in the parent, and neither survives a fork. Instead, the first use of the keys in each process starts a daemon timer that re-reads and re-unwraps the keysets every reunwrap_seconds, so a revoked KMS grant takes effect without a restart. A forked worker resets the inherited timer state and starts its own. If a re-unwrap fails, the default is to keep the cached keys and log the error, favouring availability during a KMS outage. With revoke_on_reunwrap_failure = True the cached keys are dropped and every later use must unwrap through KMS again, so revoking KMS access stops decryption within one interval, at the cost of failing requests while KMS is unreachable.

Testing your app

Set ENV=test and point the settings at throwaway keysets from hyperscale.crypto.testing:

# conftest.py
import os
from collections.abc import Iterator
from pathlib import Path

import pytest

from hyperscale.crypto import keysets, testing

os.environ.setdefault("ENV", "test")


@pytest.fixture(autouse=True)
def crypto(tmp_path: Path, settings) -> Iterator[None]:
    testing.write_dev_keysets(tmp_path)  # cleartext dev/aead.json, dev/daead.json
    settings.HYPERSCALE_CRYPTO = {"mode": "cleartext", "keyset_dir": tmp_path}
    keysets.reset()
    yield
    keysets.reset()

To exercise the KMS code path without AWS, use uri = testing.fake_kms_uri() and testing.write_kms_keysets(tmp_path, uri), with {"mode": "kms", "kms_key_uri": uri, "keyset_dir": tmp_path}. keysets.reset() drops the cached keysets so each test sees its own.

Security notes

• Deterministic encryption leaks equality. Equal plaintexts in the same column give equal ciphertexts, so anyone with database access can see which rows share a value and how often. On a unique column there is nothing to count, which is why deterministic fields require unique=True. Use them only where you must look rows up by value.
• Associated data. Every ciphertext is bound to <app_label>.<model_name>.<field> (plus |<context> when set). A value copied to another column, model or context fails to decrypt instead of being read in the wrong place.
• Only Ciphertext values pass through. A value is stored without encryption only when it is a hyperscale.crypto.fields.Ciphertext, the marker that .values(), .values_list() and unread loaded fields carry. Every plain str, including one that starts with hc1:, is plaintext and is encrypted. A ciphertext copied from another row and submitted as text is stored encrypted as that text; it never decrypts to the other row's value.
• The canary stops a process from serving with keys that cannot read the data (see Startup).

Limitations

• One database. Keyset state, the canary and the rotation log live in the default database, and rotation reads and writes models there; a using argument or database router is not honoured.
• PostgreSQL or sqlite. Other backends are untested. The share and table locks that serialise deterministic writes against a daead switch exist only on PostgreSQL; on sqlite the database's own write lock serialises them.
• dumpdata emits plaintext. Serialisation reads fields through the model, so fixtures and dumps contain decrypted values. Treat them as sensitive. loaddata re-encrypts on the way in, but cannot load a field with a context (raw saves have no instance for it).
• keyset_status / rotation.status() write the canary on first run if it does not exist yet, like startup does.
• Where each command runs. Every command records its KeyRotationEvent in the database it is pointed at, so run them against the database whose evidence you want. keyset_retire and keyset_destroy read row counts and the retire date from the production database and edit the keyset files, so they need both the production database and a checkout of the repository (commit the changed file afterwards). keyset_rotate and keyset_add_key edit the keyset files too; keyset_rotate --keyset daead also rewrites production rows.

Optional: background tasks

hyperscale.crypto.tasks.reencrypt_aead_task runs keyset_reencrypt as a background task. It uses Django's built-in tasks framework (django.tasks, Django 6.0 and later) and, on older Django, the django-tasks backport if it is installed. With neither, the module defines nothing. Configure a task backend in TASKS as usual.

from hyperscale.crypto.tasks import reencrypt_aead_task

reencrypt_aead_task.enqueue(batch_size=500, actor="ops")

Development

Requires uv and Python 3.14+.

uv sync                             # create the venv and install dependencies
uv run pytest                       # run the tests
uv run pre-commit run --all-files   # lint, format and lockfile checks

See CONTRIBUTING.md for the full set of checks.

Releasing

Bump version in pyproject.toml and __version__ in src/hyperscale/crypto/__init__.py, update CHANGELOG.md, then publish a GitHub release tagged v<version>. The publish workflow checks the tag matches the package version, builds, runs twine check and publishes to PyPI with trusted publishing.

Before the first release, register the project on PyPI with this repository and publish.yml as a trusted publisher, and create a pypi environment in the repository settings.

Design

Design spec: docs/superpowers/specs/2026-09-24-encryption-primitive-design.md

License

MIT. See LICENSE.