Python collections that are backended by sqlite3 DB and are compatible with the built-in collections
Project description
sqlitecollections
sqlitecollections
is a sort of containers that are backended by sqlite3 DB and are compatible with corresponding built-in collections. Since containers consume disk space instead of RAM, they can handle large amounts of data even in environments with limited RAM. Migrating from existing code using the built-in container is as simple as importing the library and changing the constructor.
The elements of the container are automatically serialized and stored in the sqlite3 database, and are automatically read from the sqlite3 database and deserialized when accessed. Current version supports List (mutable sequence), Dict (mutable mapping) and Set (mutable set) and almost all methods are compatible with list, dict and set respectively.
Installation
pip install sqlitecollections
Example
import sqlite3
from sqlitecollections import List, Set, Dict
conn = sqlite3.connect("collections.db")
l = List[str](
connection=conn,
table_name="list_example",
data=["Alice", "Bob", "Carol"]
)
print(l[2])
#> Carol
print(len(l))
#> 3
l.append("Dave")
print(l.index("Bob"))
#> 2
d = Dict[str, str](
connection=conn,
table_name="dict_example",
data={"a": "Alice", "b": "Bob"}
)
print(d["a"])
#> Alice
d["c"] = "Carol"
print(list(d.keys()))
#> ['a', 'b', 'c']
print(list(d.values()))
#> ['Alice', 'Bob', 'Carol']
s = Set[str](
connection=conn,
table_name="set_example",
data=["Alice", "Bob", "Carol", "Dave"]
)
print("Ellen" in s)
#> False
print("Alice" in s)
#> True
print(list(s.intersection(["Alice", "Carol"])))
#> ['Alice', 'Carol']
The database is updated with each operation, so even if we exit from the python process at this point, the database will still be in the same state and the next time we use the same file, we will be able to use the container from the last time we terminated.
import sqlite3
from sqlitecollections import List
conn = sqlite3.connect("collections.db")
l = List[str](
connection=conn,
table_name="list_example",
)
print(len(l))
#> 4
print(l[2])
#> Carol
Pros and cons for built-in containers
Pros
- Save memory usage.
- Once the database is built, loading time is almost zero, even for huge data.
Cons
- Each operation has the overhead of serialize/deserialize.
- Some operations are incompatible and unavailable. For example, directly rewriting the mutable elements of a container.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for sqlitecollections-0.5.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 48d053af1daa5ca0cff286bb48179537614d7f90c3b904fcd6af87001ca34618 |
|
MD5 | be349cc514afab28f8bddb1b9be9914f |
|
BLAKE2b-256 | 5e23e5047da4322b446dbd6293dc43daa3f4c2a6daef2e601834b7eeb7c08618 |