Python Object Storage

Simple fast JSON file storage for Python dataclasses and Pydantic models, thread and multiprocess safe.

It's standard to use SQL or NoSQL database servers as data backend, but sometimes it's more convenient to have data persisted as file(s) locally on backend application side. If you still need to use SQL for data retrieval the best option is SQLite, but for simple REST APIs it could be better to work with objects as is. So here we go.

Installation

pip install pysdato

If you plan to use it with pydantic:

pip install pysdato[pydantic]

To use with dataclasses:

pip install pysdato[dataclass]

To use with msgspec:

pip install pysdato[msgspec]

Usage

The library is intended to store Python dataclasses, msqspec.Struct or Pydantic models as JSON-files referenced by ID and supports object hierarchy.

Let's say we have Author model. Object's ID is key point for persistence -- it will be used as name of file to store and load. We can have ID as object's field, but we may also keep it outside. The default expected name of ID field is id, but it can be changed with id_field parameter of @saveable decorator: @saveable(id_field='email').

from dataclasses import dataclass
import pys

# Initialize storage with path where files will be saved
storage = pys.storage('storage.db')

@pys.saveable
@dataclass
class Author:
    name: str

# Persist model Author
leo = Author(name='Leo Tolstoy')
storage.save(leo)  # At this point the file `.storage/Author/<random uuid id>.json` will be saved
                   # with content {"name":"Leo Tolstoy"}

# Load model Author by its ID and check it's the same
another_leo = storage.load(Author, leo.__my_id__())
assert another_leo.name == leo.name

Work with dependant data

We may have a class that relates to other classes (like Authors and their Books). We can persist that dependant class separately (as we did before with Author), but we can also persist in context of their "primary" class.

import pys
from pydantic import BaseModel

# An author
@pys.saveable
class Author(BaseModel):
    name: str

# And a book
@pys.saveable
class Book(BaseModel):
    title: str

storage = pys.storage('storage.db')

# A few books of Leo Tolstoy
leo = Author(name='Leo Tolstoy')
war_and_peace = Book(title='War and peace')

# Save Leo's book
storage.save(leo)
storage.save(war_and_peace, leo)

# One more author :)
gpt = Author(name='Chat GPT')

# Do we have the same book by GPT?
gpt_war_and_peace = storage.load(Book, war_and_peace.__my_id__(), gpt)
assert gpt_war_and_peace is None

# Now it has :)
storage.save(war_and_peace, gpt)
gpt_war_and_peace = storage.load(Book, war_and_peace.__my_id__(), gpt)
assert gpt_war_and_peace is not None

We may have as many dependant models as we need. Actually, it's the way to have model dependent indexes that let us easily get (dependent) model list by another model.

import pys
from pys.pydantic import ModelWithID

# An author
class Author(ModelWithID):
    name: str

# And a book
class Book(ModelWithID):
    title: str

storage = pys.storage('storage.db')

# A few books of Leo Tolstoy
leo = Author(name='Leo Tolstoy')
war_and_peace = Book(title='War and peace')
for_kids = Book(title='For Kids')

storage.save(leo)
storage.save(war_and_peace, leo)
storage.save(for_kids, leo)

leo_books = list(storage.list(Book, leo))
assert len(leo_books) == 2
assert war_and_peace in leo_books
assert for_kids in leo_books

Embedded types

If you dataclass contains fields of some custom types (i.e. other dataclasses or types like date) they can be automatically converted with help of dacite library the pys uses for dataclasses now.

In case of primitive fields you don't need to specify anything extra:

@pys.saveable(field_as_id='some_other')
@dataclass
class A:
    some_other: str

@pys.saveable
@dataclass
class B:
    id: str

@pys.saveable
@dataclass
class C:
    name: str

@pys.saveable
@dataclass
class D:
    a: A
    b: B
    c: C

But for other types like datetime.date you need to specify type_hooks parameter of @saveable:

@pys.saveable(field_as_id='date', type_hooks={
    datetime.date: datetime.date.fromisoformat,
})
@dataclass
class D:
    date: datetime.date

More samples

Please check tests/test_samples.py for more saveable class definitions and operations.

Storages

Library supports two storages implementation:

• sqlite_storage() - SQLite based -- really fast, uses one file for all objects. Good for single process access with best performance.
• file_storage() - JSON file per object storage, it is slower, but saves each object in a separate JSON file. Multiprocess- and thread-safe, but can make FS DoS with too many objects.
• zip_storage() - ZIP-file based -- slow, compact, uses one file for all objects. Multiprocess- and thread-safe, compact file storage.
• in_memory_storage(parent=<any storage>) - In-memory storage -- very fast, compact, stores one object via given parent storage. Multiprocess- and thread-safe depends on parent storage (file_storage is recommended.)

The default storage is in_memory_storage based on a file_storage.

Library Reference

import pys

# Initialize file storage
storage = pys.file_storage('.path-to-storage')

# Initialize default (in memory) storage
storage = pys.storage()

# Initialize SQLite storage
storage = pys.sqlite_storage('path-to-storage.db')

# Initialize ZIP-file storage
storage = pys.zip_storage('path-to-storage.zip')

# Initialize in-memory storage with file storage backend
storage = pys.in_memory_storage(parent=file_storage('.mem'))

# Save a model with optional relation to other models
storage.save(model, [related_model | (RelatedModelClass, related_model_id), ...])

# Load a model by ModelClass and model_id with optional relation to other models
storage.load(ModelClass, model_id, [related_model | (RelatedModelClass, related_model_id), ...])

# Delete a model by ModelClass and model_id with optional relation to other models
storage.delete(ModelClass, model_id, [related_model | (RelatedModelClass, related_model_id), ...])

# List models by specified ModelClass with optional relation to other models
storage.list(ModelClass, [related_model | (RelatedModelClass, related_model_id), ...])

# Destroy storage
storage.destroy()

Benchmark

You can find the benchmark code in benchmark.py file.

Storage: file.Storage(base_path=benchmark.storage)
T1: 596.98 ms -- save 1100 objects -- 0.543 ms per object
T2: 1218.77 ms -- list 500 objects -- 2.438 ms per object
T3: 979.78 ms -- list 500 objects -- 1.960 ms per object
Storage: sqlite.Storage(base_path=benchmark.db)
T1: 10.03 ms -- save 1100 objects -- 0.009 ms per object
T2: 0.00 ms -- list 500 objects -- 0.000 ms per object
T3: 0.00 ms -- list 500 objects -- 0.000 ms per object
Storage: file.Storage(base_path=benchmark.zip)
T1: 23195.79 ms -- save 1100 objects -- 21.087 ms per object
T2: 2131.86 ms -- list 500 objects -- 4.264 ms per object
T3: 1534.07 ms -- list 500 objects -- 3.068 ms per object
Storage: in_memory.Storage(parent=file.Storage(base_path=.mem))
T1: 710.24 ms -- save 1100 objects -- 0.646 ms per object
T2: 16.09 ms -- list 500 objects -- 0.032 ms per object
T3: 0.00 ms -- list 500 objects -- 0.000 ms per object

Release Notes

• 0.0.17 Added support for embedded initialization for @dataclass
• 0.0.16 Fixed in-memory storage persistence bugs.
• 0.0.15 In-memory storage with any persistence backend is added.
• 0.0.14 ZIP-file based storage is added.
• 0.0.13 ID can be any type.
• 0.0.12 Fixed: issue with file encoding for custom raw models.
• 0.0.11 Fixed: use own __my_id__() function if defined in data class.
• 0.0.10 Minor changes in documentation.
• 0.0.9 improved performance, generic Persistent base class is provided for custom implementations, allowed installing specifically for pydantic, dataclasses or msgspec usage.
• 0.0.8 unit-test covers more cases now. Object's actual ID can be used even if it's not defined. Documentation is updated.
• 0.0.7 build and test for different Python versions.
• 0.0.6 saveable decorator reworked, added default_id parameter that can be used for changing ID generation behaviour. By default, we use str(uuid.uuid4()) as ID.
• 0.0.5 Performance is dramatically improved with SQLite storage implementation. Default storage is SQLite storage now.
• 0.0.4 SQLite storage is added. Support of msqspec JSON and structures is added.
• 0.0.3 Benchmark is added, performance is improved. Fixed dependency set up.
• 0.0.2 Added support for Python 3.x < 3.10
• 0.0.1 Initial public release