Simple JSON file storage for Python dataclasses and pydantic models, thread and multiprocess safe
Project description
Python Object Storage
Simple fast JSON file storage for Python dataclasses and Pydantic models, thread and multiprocess safe.
Installation
pip install pysdato
Usage
The library is intended to store Python dataclasses or Pydantic models as JSON-files referenced by ID and supports object hierarchy.
Let's say we have Author
model. Object's ID is key point for persistence -- it will be used as name of
file to store and load. We can have ID as object's field, but we may also keep it outside.
The default expected name of ID field is id
, but it can be changed with id_field
parameter of @saveable
decorator: @saveable(id_field='email')
.
from dataclasses import dataclass
import pys
# Initialize storage with path where files will be saved
storage = pys.storage('.storage')
@pys.saveable
@dataclass
class Author:
name: str
# Persist model Author
leo = Author(name='Leo Tolstoy')
leo_id = storage.save(leo) # At this point the file `.storage/Author/<random uuid id>.json` will be saved
# with content {"name":"Leo Tolstoy"}
# Load model Author by its ID and check it's the same
another_leo = storage.load(Author, leo_id)
assert another_leo.name == leo.name
Work with dependant data
We may have a class that relates to other classes (like Authors and their Books). We can persist
that dependant class separately (as we did before with Author
), but we can also persist
in context of their "primary" class.
import pys
from pydantic import BaseModel
# An author
@pys.saveable
class Author(BaseModel):
name: str
# And a book
@pys.saveable
class Book(BaseModel):
title: str
storage = pys.storage('.storage')
# A few books of Leo Tolstoy
leo = Author(name='Leo Tolstoy')
war_and_peace = Book(title='War and peace')
# Save Leo's book
leo_id = storage.save(leo)
wnp_id = storage.save(war_and_peace, leo)
# One more author :)
gpt = Author(name='Chat GPT')
# Do we have the same book by GPT?
gpt_war_and_peace = storage.load(Book, wnp_id, gpt)
assert gpt_war_and_peace is None
# Now it has :)
storage.save(war_and_peace, gpt)
gpt_war_and_peace = storage.load(Book, wnp_id, gpt)
assert gpt_war_and_peace is not None
We may have as many dependant models as we need. Actually, it's the way to have model dependent indexes that let us easily get (dependent) model list by another model.
from pydantic import BaseModel
import pys
# An author
@pys.saveable
class Author(BaseModel):
name: str
# And a book
@pys.saveable
class Book(BaseModel):
title: str
storage = pys.storage('.storage')
# A few books of Leo Tolstoy
leo = Author(name='Leo Tolstoy')
war_and_peace = Book(title='War and peace')
for_kids = Book(title='For Kids')
storage.save(leo)
storage.save(war_and_peace, leo)
storage.save(for_kids, leo)
leo_books = list(storage.list(Book, leo))
assert len(leo_books) == 2
assert war_and_peace in leo_books
assert for_kids in leo_books
Reference
import pys
# Initialize default (file) storage
storage1 = pys.storage('.path-to-storage')
# Initialize SQLite storage
storage2 = pys.sqlite_storage('path-to-storage.db')
# Save a model with optional relation to other models
storage.save(model, [related_model | (RelatedModelClass, related_model_id), ...])
# Load a model by ModelClass and model_id with optional relation to other models
storage.load(ModelClass, model_id, [related_model | (RelatedModelClass, related_model_id), ...])
# Delete a model by ModelClass and model_id with optional relation to other models
storage.delete(ModelClass, model_id, [related_model | (RelatedModelClass, related_model_id), ...])
# List models by specified ModelClass with optional relation to other models
storage.list(ModelClass, [related_model | (RelatedModelClass, related_model_id), ...])
# Destroy storage
storage.destroy()
Release Notes
0.0.4
SQLite storage is added.
Support of msqspec
JSON and structures is added.
0.0.3
Benchmark is added, performance is improved. Fixed dependency set up.
0.0.2
Added support for Python 3.x < 3.10
0.0.1
Initial public release
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.