django-data-shape

A realistically shaped test database from Django models.

Declare the shape of your data -- cardinality, value skew, foreign-key fan-out as a distribution with a long tail, and where related rows physically sit -- then load it by COPY and ANALYZE it, so the query planner makes the same choices it will make in production.

It exists because a plan over ten rows is a lie, and because the loop it replaces is not merely smaller: uniform fan-out makes the planner always right, and generating children parent-by-parent clusters them perfectly, which flatters every index scan. A test database can be wrong in the flattering direction, and usually is.

Install

pip install 'django-data-shape[postgres]'

The quotes are not decoration: zsh globs the brackets and reports no matches found without them.

Use

import datetime

from django_data_shape import Sequential, Shape, Skew, Table, Uniform, build

from myapp.models import Order

shape = Shape(
    Table(
        Order,
        rows=1_000_000,
        status=Skew({"complete": 0.98, "pending": 0.015, "cancelled": 0.005}),
        total=Uniform(0, 500, places=2),
        created_at=Sequential(
            datetime.datetime(2020, 1, 1, tzinfo=datetime.timezone.utc),
            datetime.timedelta(seconds=3),
        ),
    ),
    seed=1234,
)

build(shape)

build() generates the rows, loads them with COPY, moves the identity sequence past the keys it assigned, and runs ANALYZE so the planner can see the shape. It raises on any backend that is not PostgreSQL rather than degrading quietly.

Relations

from django_data_shape import Constant, FanOut, Shape, Table, Zipf, build

build(
    Shape(
        Table(Company, rows=50, name=Constant("acme")),
        Table(
            Order,
            rows=2_000_000,
            # A distribution, not a number: giving every parent ten children is
            # the one shape in which the planner is never wrong, because its
            # n_distinct average is then the truth.
            company=FanOut(Zipf(1.2), childless=0.35),
            status=Constant("complete"),
        ),
    )
)

The parents can be rows this package built or rows your own code did -- their real keys are read, not assumed, so the ORM can own the small tables while this owns the large ones.

What it expects, and what it refuses

A declaration that cannot describe a database raises before a row is generated, naming the field. In particular:

• PostgreSQL and psycopg 3. Rows stream into COPY FROM STDIN, which psycopg 2 cannot do without materialising them first. Both are refused by name rather than degraded around.
• A key type it can assign. Integer keys count from one and UUID keys are derived from the seed; anything else is refused rather than guessed, and keys=KeyFunction(...) declares one.
• Empty tables. Keys start at 1 on every build, so build() checks first and raises rather than colliding partway through.
• A callable model default such as default=uuid4 must be declared as a distribution: uuid4 varies per row and dict does not, and nothing on the field distinguishes them.

Status

Early. Single tables and the model graph. Derived fields, collections copied along a join, per-group invariants and template-database reuse come next.

Full documentation: https://artui.github.io/django-data-shape/

License

MIT