sqlitecollections
sqlitecollections is a sort of containers that are backended by sqlite3 DB and are compatible with corresponding built-in collections. Since containers consume disk space instead of RAM, they can handle large amounts of data even in environments with limited RAM. Migrating from existing code using the built-in container is as simple as importing the library and changing the constructor.
The elements of the container are automatically serialized and stored in the sqlite3 database, and are automatically read from the sqlite3 database and deserialized when accessed. Current version supports List (mutable sequence), Dict (mutable mapping) and Set (mutable set) and almost all methods are compatible with list, dict and set respectively.
Installation
pip install sqlitecollections
Example
import sqlitecollections as sc
l = sc.List[str](["Alice", "Bob", "Carol"])
print(l[2])
#> Carol
print(len(l))
#> 3
l.append("Dave")
print(l.index("Bob"))
#> 1
print(l.index("Dave"))
#> 3
d = sc.Dict[str, str]({"a": "Alice", "b": "Bob"})
print(d["a"])
#> Alice
d["c"] = "Carol"
print(list(d.keys()))
#> ['a', 'b', 'c']
print(list(d.values()))
#> ['Alice', 'Bob', 'Carol']
s = sc.Set[str](["Alice", "Bob", "Carol", "Dave"])
print("Ellen" in s)
#> False
print("Alice" in s)
#> True
print(list(s.intersection(["Alice", "Carol"])))
#> ['Alice', 'Carol']
In the above example, a temporary file is created every time a container is created, and the elements are written to the sqlite3 database created on the file, thus consuming very little RAM.
If you want to reuse the container you created, you can create it by specifying the file path and table name of the sqlite3 database.
import sqlitecollections as sc
l = sc.List[str](["Alice", "Bob", "Carol"], connection="path/to/file.db", table_name="list_example")
l.append("Dave")
exit()
When you load it, you can restore the previous state by specifying the same file path and table name.
import sqlitecollections as sc
l = sc.List[str](connection="path/to/file.db", table_name="list_example")
print(len(l))
#> 4
print(list(l))
#> ['Alice', 'Bob', 'Carol', 'Dave']
Pros and cons for built-in containers
Pros
- Save memory usage.
- Once the database is built, loading time is almost zero, even for huge data.
Cons
- Each operation has the overhead of serialize/deserialize.
- Some operations are incompatible and unavailable. For example, directly rewriting the mutable elements of a container.
1 maintainer
Metadata
1 maintainer
Release files for sqlitecollections 1.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| sqlitecollections-1.2.0.tar.gz | 18.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sqlitecollections-1.2.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 39.1 kB
Release files / sqlitecollections-1.2.0.tar.gz
| Download URL | sqlitecollections-1.2.0.tar.gz |
|---|---|
| Size | 18.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b08e3387763014a64b42cf9bb8afbf19c6c0ab54c48ba2e726ac6a3ad22f9d11
|
|
BLAKE2b-256 checksum How to use checksums |
368932563487771926e2a5285537d75703abb6408f4b240f3d51d9810c038c12
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.8.14
|
Release files / sqlitecollections-1.2.0-py3-none-any.whl
| Download URL | sqlitecollections-1.2.0-py3-none-any.whl |
|---|---|
| Size | 21.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7cf789d3f3c1a54283c7a53da2b2405ceaaa57908ebc2d925fc4f32499359d5d
|
|
BLAKE2b-256 checksum How to use checksums |
ca4f703a6ba421164f363bb4ac7ed9e03276e5ca9c16a3c6353680674dd47a43
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.8.14
|
