inmemory-datastore-stub

An in-memory Google Cloud Datastore stub for unit-testing python-ndb code — no emulator, no Java, no network, no gcloud component to install.

It implements the Datastore RPC surface in pure Python, so your tests run at in-process speed.

Why this package exists

This is a maintained fork of InMemoryCloudDatastoreStub by Phil Lopreiato, which has been unmaintained since January 2021 and only works with google-cloud-datastore 1.x. On the 2.x libraries it fails outright:

• google.cloud.datastore_v1.proto was removed in datastore 2.x, so importing the original raises ModuleNotFoundError
• datastore 2.x wraps every message in proto-plus, which does not expose the protobuf-only API the stub relies on (WhichOneof, HasField, CopyFrom, SerializeToString)
• ndb 2.x calls the RPC methods by snake_case name (lookup, commit, run_query, …) while the original only exposes the 1.x CamelCase names
• ndb 2.x expects a database attribute on the client
• allocate_ids was never implemented

This fork fixes all of the above and is tested on Python 3.10–3.14.

Install

pip install inmemory-datastore-stub

Usage

The simplest form — use the stub-backed client directly:

from google.cloud import ndb
from inmemory_datastore_stub import Client


class Movie(ndb.Model):
    title = ndb.StringProperty()


def test_movies():
    with Client().context():
        Movie(title="Roman Holiday").put()
        assert Movie.query().count() == 1

As a pytest fixture:

import pytest
from inmemory_datastore_stub import Client


@pytest.fixture
def ndb_context():
    with Client().context():
        yield

If the code under test constructs its own ndb.Client(), patch it globally:

from inmemory_datastore_stub import patch_ndb

with patch_ndb():
    import myapp  # myapp calls ndb.Client() at import time

    myapp.run()

patch_ndb() restores the original ndb.Client on exit.

Catching bugs the stub would otherwise hide

Two behaviours of real Datastore make code fail in production that passes against a naive in-memory stub. Both are reproduced here, and both are opt-in so they never break an existing suite.

Eventual consistency

Non-ancestor queries in Datastore do not see writes immediately. Turn that on and the stub holds fresh writes back until you call catch_up():

client = Client(eventual_consistency=True)

with client.context():
    Movie(title="fresh").put()

    assert Movie.query().fetch() == []  # non-ancestor query lags, as in production
    assert movie.key.get().title == "fresh"  # key lookups are strongly consistent

    client.catch_up()
    assert len(Movie.query().fetch()) == 1

Ancestor queries and key lookups stay strongly consistent, matching Datastore.

Composite index validation

Datastore refuses any query whose composite index is not declared in index.yaml. Point the stub at yours and it refuses them too — with the same index suggestion Datastore gives you, ready to paste:

client = Client(index_yaml="index.yaml")

NoMatchingIndexError: no matching index found. recommended index is:
indexes:
- kind: Movie
  properties:
  - name: genre
  - name: title

Needs PyYAML: pip install 'inmemory-datastore-stub[indexes]'.

Size limits

Datastore rejects entities that are too large, and the failure only shows up once real data arrives. These limits are enforced on write, so the test fails where the bug is:

Doc(blob="x" * 1_500_000).put()
# LimitExceededError: entity is 1500051 bytes, over the 1048572 byte limit

Doc(tags=["t"] * 21_000).put()
# LimitExceededError: entity has 21000 index entries, over the 20000 limit

Enforced: entity size (1 MiB - 4 bytes), key size (6 KiB), indexed string values (1500 bytes), nested value depth (20), index entries per entity (20,000), and keys per lookup (1000).

LimitExceededError subclasses google.api_core.exceptions.InvalidArgument, which is what Datastore raises, so an except InvalidArgument in production code catches it here too. This is the one behaviour that is on by default — pass enforce_limits=False to store what the real service would refuse.

What is supported

put, get, delete, put_multi/get_multi/delete_multi, get_or_insert, allocate_ids, transactions, and ancestor queries.

Queries: equality, inequality, !=, IN, NOT_IN, OR, repeated properties, StructuredProperty and ComputedProperty, ordering by several properties in either direction, offset/limit, count, cursors and fetch_page, projection and keys-only queries, and GQL.

Not reproduced, by design: index build state lives in the Admin API rather than index.yaml; query planning has no meaning without a real planner, and inventing ExplainMetrics numbers would be worse than omitting them; write rate limits need load and wall-clock time, which unit tests do not have. For those, use the official Datastore emulator.

License

MIT — see LICENSE. Original work © 2020 Phil Lopreiato; modifications © 2026 Stepan Shamaiev.