IC Python DB
A lightweight key-value database with entity relationships and audit logging capabilities, intended for small to medium-sized applications running on the Internet Computer. Forked from kybra-simple-db.
Features
- Persistent Storage: Works with StableBTreeMap stable structure for persistent storage on your canister's stable memory so your data persists automatically across canister upgrades.
- Range-based paging:
page,load_someandrebuild_field_indexread a page of entities with a single orderedrangecall on the stable map (BasiliskStableBTreeMap.range), so cost tracks the rows returned rather than the highest id ever issued, and deleted ids are free. Backends withoutrange(older CDK builds, plain dict test doubles) transparently fall back to one read per id.instances()andfind()load a whole type: they are deprecated and raiseTooLargeErrorpast 1,000 entities (usecount(),find_by()orpage()). - Entity-Relational Database: Create, read and write entities with OneToOne, OneToMany, ManyToOne, and ManyToMany relationships.
- Schema Versioning & Upgrade Safety: Automatic schema introspection, compatibility checking, and auto-migration for safe changes. Breaking changes without a
migrate()method are rejected, preventing data corruption on canister upgrades. - Entity Hooks: Intercept and control entity lifecycle events (create, modify, delete) with
on_eventhooks. - Access Control: Thread-safe context management for user identity tracking and ownership-based permissions.
- Namespaces: Organize entities into namespaces to avoid type conflicts when you have multiple entities with the same class name.
- Audit Logging: Track all changes to your data with created/updated timestamps and who created and updated each entity.
Installation
pip install ic-python-db
Quick Start
The database storage must be initialized before using IC Python DB. Here's an example of how to do it:
from basilisk import StableBTreeMap
from ic_python_db import Database
# Initialize storage and database
storage = StableBTreeMap[str, str](memory_id=1, max_key_size=100, max_value_size=1000) # Use a unique memory ID for each storage instance
Database.init(db_storage=storage)
Read Basilisk's documentation for more information regarding StableBTreeMap and memory IDs.
Next, define your entities:
from ic_python_db import (
Database, Entity, String, Integer,
OneToOne, OneToMany, ManyToOne, ManyToMany, TimestampedMixin
)
class Person(Entity, TimestampedMixin):
__alias__ = "name" # Use `name` as the alias field for lookup
name = String(min_length=2, max_length=50)
age = Integer(min_value=0, max_value=120)
friends = ManyToMany("Person", "friends")
mother = ManyToOne("Person", "children")
children = OneToMany("Person", "mother")
spouse = OneToOne("Person", "spouse")
Entity Lookup
Entities can be retrieved using Entity[key] syntax with three different lookup modes:
# Create an entity
john = Person(name="John", age=30)
# Lookup by ID
Person[1] # Returns john (by _id)
Person["1"] # Also works with string ID
# Lookup by alias (defined via __alias__)
Person["John"] # Tries ID first, then alias field "name"
# Lookup by specific field (tuple syntax)
Person["name", "John"] # Lookup by field "name" only
| Syntax | Behavior |
|---|---|
Person[1] |
Lookup by _id |
Person["John"] |
Try _id first, then __alias__ field |
Person["name", "John"] |
Lookup by specific field name only |
Then use the defined entities to store objects:
# Create and save an object
john = Person(name="John", age=30)
# Update an object's property
john.age = 33 # Type checking and validation happens automatically
# use the `_id` property to load an entity with the [] operator
Person(name="Peter")
peter = Person["Peter"]
# Delete an object
peter.delete()
# Create relationships
alice = Person(name="Alice")
eva = Person(name="Eva")
john.mother = alice
assert john in alice.children
eva.friends = [alice]
assert alice in eva.friends
assert eva in alice.friends
print(alice.serialize()) # Prints the dictionary representation of an object
# Prints: {'timestamp_created': '2025-09-12 22:15:35.882', 'timestamp_updated': '2025-09-12 22:15:35.883', 'creator': 'system', 'updater': 'system', 'owner': 'system', '_type': 'Person', '_id': '3', 'name': 'Alice', 'age': None, 'children': '1', 'friends': '4'}
assert Person.count() == 3
assert Person.max_id() == 4
# Cursor-based pagination: (page, next_from_id), None on the last page
page, cursor = Person.page(1, 2)
assert page == [john, alice]
page, cursor = Person.page(cursor, 2)
assert page == [eva] and cursor is None
# Retrieve database contents in JSON format
print(Database.get_instance().dump_json(pretty=True))
# Audit log
audit_records = Database.get_instance().get_audit(id_from=0, id_to=5)
pprint(audit_records['0'])
''' Prints:
['save',
1744138342934,
'Person@1',
{'_id': '1',
'_type': 'Person',
'age': 30,
'creator': 'system',
'name': 'John',
'owner': 'system',
'timestamp_created': '2025-04-08 20:52:22.934',
'timestamp_updated': '2025-04-08 20:52:22.934',
'updater': 'system'}]
'''
For more usage examples, see the tests.
Indexed Fields
Declare indexed=True on a scalar property to maintain a persistent secondary index, enabling equality lookups without scanning all entities:
class Proposal(Entity):
title = String()
status = String(max_length=32, indexed=True)
org_scope = String(max_length=128, indexed=True)
Proposal(title="A", status="voting", org_scope="Justice")
Proposal(title="B", status="executed", org_scope="Justice")
# Paginated equality lookup — cost proportional to matches, not max_id
entities, next_from_id = Proposal.find_by("status", "voting", from_id=1, count=50)
# Count without loading entities
Proposal.count_by("org_scope", "Justice") # 2
The index is maintained automatically on create, update, and delete. None values are not indexed. Querying a field that is not declared indexed=True raises ValueError.
Backfilling existing data
Entities created before a field became indexed are missing from its index. Rebuild in bounded batches (resumable, safe for IC instruction limits — spread calls across multiple update calls or timers):
cursor = 1
while cursor is not None:
cursor = Proposal.rebuild_field_index("status", from_id=cursor, batch=200)
The rebuild is idempotent. The indexed flag is not part of the schema descriptor, so toggling it never triggers upgrade-compatibility errors.
Each matching id is its own storage entry, so a value can match any number of entities without reaching the backing store's max_value_size.
Values that hold |
An index entry is a key that starts with the value, then |, a letter and the id (_fi:Ticket:status:open|n000000000042). A value is text a caller chooses, and one that ended like such a key would put keys of its own among the entries of another value, where a page read could miss some of them. So a value with a | in it is escaped (| as %7C, and % as %25, so that two values never share an escaped form) and filed under _fx instead of _fi: no value can hold a |, so the keys of two values never meet. Every other value keeps the key it always had, one with only a % in it included. The escaped key is longer (two more characters for each | or %), which counts against the storage map's max_key_size.
Upgrading from 0.14. Entries of values with a |, filed before this under _fi with the value as it was, stay readable: find_by, count_by and field_index_* also read the old key of such a value, and adding or removing an id takes it off the old list. If 0.13 or 0.14 already ran on a store of 0.12 that holds the old lists of both a value V and V|n and 12 digits, or V|h and the digest of a custom id in V's list, converting V wrote its entry over the other value's old list: that value's ids are lost. Entity.rebuild_field_indexes re-adds the sequential ones (run it, then the repair); a custom id comes back only when its entity's field is next changed, or when Database.field_index_add is called for it. To move them all, run once after upgrading, in bounded calls:
db = Database.get_instance()
cursor = ""
while cursor is not None:
cursor = db.repair_field_indexes(cursor, limit=500) # keys read per call
repair_field_indexes checks each entry of a value with a | against its entity: it moves an old entry to its present key and removes one whose entity is gone or has another value. It reaches custom ids and keys of any length, and it converts a list that a release before 0.13 stored as one array under the old key of such a value (a list counts its ids against limit; one longer than limit is converted whole, alone in its call). Nothing needs to run before it. Entity.rebuild_field_indexes(from_id, batch) does the move per type for sequential ids only, and adds entries that are missing; it never removes one.
When the repair has run to its end no old key holds an entry and counts are exact, except for what it cannot judge and leaves as it is: entries of a type or field the code does not index; entries whose row has no such field; an old-form list of more than 100 thousand ids (it is logged; a read of its value converts it where it fits the message, about 160 thousand ids in an update and 20 thousand in a query, and traps where it does not, as before); and an old-form list of a value made to end with |h and the digest of its own text, which reads as an entry of the value it extends (only such a value matches; find_by is not affected). A chain of old lists of V, V|c, V|c|c and so on deeper than about 500 raises RecursionError (also a value made for it). Only then may a release drop the old-key reads.
Cost, measured on the IC: a key read about 0.1 million instructions; an entry of a value with a | that is right about 0.73 million (its row is loaded); an old entry moved about 2 million, so a call of 500 moves is about 1 billion; and about 0.58 million for each distinct key length in the whole map that a call walks past. A call is O(limit + those lengths). An old-form list is converted whole at about 245 thousand instructions an id: a map with limited value size holds at most about 10 thousand ids in a list, but a realm's is not limited (a list of 127,000 ids is on record: about 31 billion instructions; it is over the cap of 100 thousand ids, so the repair leaves it, and a read of its value in an update converts it).
Rolling back to a release before this one, and upgrading again. While the older release runs, it files values with a | under _fi (this release reads them there), but it does not know _fx: entries filed under it are invisible to it, and it does not remove one when an entity's value changes or the entity is deleted. After the next upgrade a list can hold an entity that no longer has the value, or an id on both lists, so count_by and find_by can be wrong for those values. Run repair_field_indexes to its end again, from after="": a cursor, or a note that it was done, saved before the rollback skips what the older release wrote. It puts it right.
Sorted Fields
Declare sorted=True on an Integer or Float property to read entities in its order, highest or lowest first, without loading and sorting them all. __sorted_indexes__ adds indexes over several fields, where each field breaks ties in the one before:
class Listing(Entity):
name = String()
installs = Integer(sorted=True)
likes = Integer(sorted=True)
updated_at = Float(sorted=True)
__sorted_indexes__ = [("installs", "likes")] # installs, then likes
top, cursor = Listing.top_by("installs", count=20) # most installed
top, cursor = Listing.top_by(("installs", "likes"), count=20) # ties broken by likes
recent, cursor = Listing.top_by("updated_at", count=20)
oldest, cursor = Listing.top_by("updated_at", count=20, descending=False)
more, cursor = Listing.top_by("installs", count=20, cursor=cursor) # next page; None at the end
The storage map keeps keys in order, so each entity has one index entry per sorted index whose key starts with its values. Writing a sorted field moves that entry (a few storage operations), and a page is one range read plus count loads, however many entities exist. Ties go to the higher id; entities with None in a field are left out of indexes on it. Integers must fit in 64 bits.
where= drops entities that fail a test. At most scan_limit entries (default 200) are read per call, so with a selective filter a page can come back short with a cursor: call again.
Fields made sorted after rows exist, or whose type changed, need a resumable rebuild (Listing.rebuild_sorted_index("installs", from_id=cursor), until it returns None). A migrate() that changes a sorted or indexed value moves the entity's entries when it is loaded.
Large Relations
OneToMany and ManyToMany getters return a RelationView, not a list. Questions about the relation and single pages never load every member, so they stay cheap however large it grows:
if section.stands: ... # one read
len(section.stands) # one read
stand in section.stands # two reads
stands, cursor = section.stands.page(1, 50) # one range read + 50 loads
group.members.add(user) # index writes only
Iterating, indexing and comparing still work, but they load every member, so past constants.MAX_UNPAGED (1,000) they raise TooLargeError instead of running into the IC instruction limit. Walk a large relation with page(), across calls if needed:
cursor = 1
while cursor is not None:
stands, cursor = section.stands.page(cursor, 50)
list(view) gives a plain list. Deleting a parent is O(1): its index entries are purged in bounded steps by later writes, or by Database.get_instance().purge_step() from a timer.
Namespaces
Organize entities with the __namespace__ attribute to avoid type conflicts when you have the same class name in different modules:
# In app/models.py
class User(Entity):
__namespace__ = "app"
name = String()
role = String()
# In admin/models.py
class User(Entity):
__namespace__ = "admin"
name = String()
permissions = String()
from app.models import User as AppUser
from admin.models import User as AdminUser
app_user = AppUser(name="Alice", role="developer") # Stored as "app::User"
admin_user = AdminUser(name="Bob", permissions="all") # Stored as "admin::User"
# Each namespace has isolated ID sequences and storage
assert app_user._id == "1"
assert admin_user._id == "1"
Entity Hooks
Intercept and control entity changes with the on_event hook:
from ic_python_db import Entity, String, ACTION_MODIFY
class User(Entity):
name = String()
email = String()
@staticmethod
def on_event(entity, field_name, old_value, new_value, action):
# Validate email format
if field_name == "email" and "@" not in new_value:
return False, None # Reject invalid email
# Auto-capitalize names
if field_name == "name":
return True, new_value.upper()
return True, new_value
user = User(name="alice", email="alice@example.com")
assert user.name == "ALICE" # Auto-capitalized
See docs/HOOKS.md for more patterns.
Access Control
User context management with as_user():
from ic_python_db import Database, Entity, String, ACTION_MODIFY, ACTION_DELETE
from ic_python_db.mixins import TimestampedMixin
from ic_python_db.context import get_caller_id
class Document(Entity, TimestampedMixin):
title = String()
@staticmethod
def on_event(entity, field_name, old_value, new_value, action):
caller = get_caller_id()
# Only owner can modify or delete
if action in (ACTION_MODIFY, ACTION_DELETE):
if entity._owner != caller:
return False, None
return True, new_value
db = Database.get_instance()
# Alice creates a document
with db.as_user("alice"):
doc = Document(title="My Doc") # Owner: alice
# Bob cannot modify Alice's document
with db.as_user("bob"):
doc.title = "Hacked" # Raises ValueError
See docs/ACCESS_CONTROL.md and examples/simple_access_control.py.
Type Hints
The library is fully typed (PEP 561 compliant). Type checkers and IDEs automatically infer property types:
class User(Entity):
name = String() # Inferred as str
age = Integer() # Inferred as int
active = Boolean() # Inferred as bool
user = User(name="Alice", age=30, active=True)
user.name # IDE knows this is str
user.age # IDE knows this is int
For stricter typing, you can add explicit annotations:
from typing import Optional
class User(Entity):
name: str = String()
age: int = Integer()
profile: Optional["Profile"] = OneToOne("Profile", "user")
Schema Versioning & Upgrade Safety
ic-python-db includes built-in upgrade compatibility checking. The system introspects your Entity class definitions to detect schema changes and ensure safe upgrades.
Auto-migration for safe changes
Adding a field with a default value requires no migration code — the system handles it automatically:
# v1
class Product(Entity):
__version__ = 1
name = String()
# v2 — just add the field with a default, no migrate() needed
class Product(Entity):
__version__ = 2
name = String()
price = Float(default=0.0) # auto-injected for existing entities
active = Boolean(default=True) # auto-injected for existing entities
Custom migration for breaking changes
For type changes, field renames, or data transformations, override migrate():
class Product(Entity):
__version__ = 2
name = String()
price_dollars = Float() # was price_cents: Integer in v1
@classmethod
def migrate(cls, obj, from_version, to_version):
if from_version == 1:
obj["price_dollars"] = obj.pop("price_cents") / 100.0
return obj
Upgrade compatibility enforcement
Call check_upgrade_compatibility() from your canister's post_upgrade to reject incompatible upgrades before they corrupt data:
from basilisk import post_upgrade
from ic_python_db import Database
@post_upgrade
def on_post_upgrade():
db = Database.get_instance()
db.check_upgrade_compatibility()
# If a breaking change lacks migrate(), this raises SchemaIncompatibleError
# which traps post_upgrade, causing the IC to roll back the upgrade
The system classifies schema changes as:
| Change | Safety | Action |
|---|---|---|
Add field with default= |
Safe | Auto-migrated |
| Remove field | Safe | Old data ignored |
| Add new Entity type | Safe | No migration needed |
| Change field type | Breaking | Requires migrate() |
| Add field without default | Breaking | Requires migrate() |
| Change relationship type | Breaking | Requires migrate() |
Schema introspection
You can inspect and compare schemas programmatically:
from ic_python_db import Database, build_schema, diff_schemas
db = Database.get_instance()
# Build current schema from Entity definitions
schema = db.build_schema_from_entities()
# Compare two schemas
changes = diff_schemas(old_schema, new_schema)
for change in changes:
print(f"{change.entity_type}.{change.field}: {change.reason}")
See docs/SCHEMA_VERSIONING.md for the full reference.
API Reference
- Core:
Database,Entity - Properties:
String,Integer,Float,Boolean - Relationships:
OneToOne,OneToMany,ManyToOne,ManyToMany - Mixins:
TimestampedMixin(timestamps and ownership tracking) - Hooks:
ACTION_CREATE,ACTION_MODIFY,ACTION_DELETE - Context:
get_caller_id(),set_caller_id(),Database.as_user() - Schema:
build_schema,diff_schemas,schema_hash,SchemaIncompatibleError
Development
Setup Development Environment
# Clone the repository
git clone https://github.com/smart-social-contracts/ic-python-db.git
cd ic-python-db
# Recommended setup
pyenv install 3.10.7
pyenv local 3.10.7
python -m venv venv
source venv/bin/activate
# Install development dependencies
pip install -r requirements-dev.txt
# Running tests
./run_linters.sh && (cd tests && ./run_test.sh && ./run_test_ic.sh)
Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
License
MIT.
Metadata
Release files for ic-python-db 0.15.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ic_python_db-0.15.0.tar.gz | 65.1 kB | Without provenance | Details for ic_python_db-0.15.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| ic_python_db-0.15.0-py3-none-any.whl | Python 3 | none | any | Details for ic_python_db-0.15.0-py3-none-any.whl |
Total release size: 125.3 kB
Release files / ic_python_db-0.15.0.tar.gz
| Download URL | ic_python_db-0.15.0.tar.gz |
|---|---|
| Size | 65.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ed62bfba03e795871f5a3564d20639c1d46b739c89f10175bbc9e1ca89760e35
|
|
BLAKE2b-256 checksum How to use checksums |
f4dfbd948fc8233ac2e51ad49c02212e42bbe86e18a7c2f895fc2dea18bf5dd9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.22
|
Release files / ic_python_db-0.15.0-py3-none-any.whl
| Download URL | ic_python_db-0.15.0-py3-none-any.whl |
|---|---|
| Size | 60.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c5f02d5300070beef444d9a94417aebb2bb386eff361f2fee46a54a01b3219f3
|
|
BLAKE2b-256 checksum How to use checksums |
b7de57dc3ec03dc21f7fa004c2bacbc7c5016fdf59ec355f8f1cb42f773947a3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.22
|
Security for ic-python-db 0.15.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.