static_ondisk_kv
Simple and fast implementation of a static on disk kv, in python
Why this lib?
leveldb, rocksdb and lmdb all have issues for a static collections of key and values:
- slow to build (many hours) : 3h for rocksdb compared to 1h for this lib (for a 5B collections for 1 long and 2 float16)
- uses more space than necessary (100GB for rocksdb unlike 60GB)
- as fast as this much simpler lib: about 5k sample/s on nvme drive
What this lib does not support:
- non static collection
- variable length values and keys
Install
pip install static_ondisk_kv
Python examples
Checkout these examples:
from static_ondisk_kv import OnDiskKV
from tqdm import tqdm
import random
kv = OnDiskKV(file='/media/nvme/mybigfile', key_format="q", value_format="ee")
print("length", kv.length)
k = kv.get_key(100)
v = kv.get_value(100)
print(k)
print(v)
print(kv[k])
API
OnDiskKV(file, key_format="q", value_format="ee")
Creates an ondisk kv from file using key_format and value_format for decoding.
get_key(i)
Returns the key at position i.
get_value(i)
Returns the value at position i.
getitem(k)
Returns the value for the key k
sort_parquet(input_collection, key_column, value_columns, output_folder)
sort parquet files of collection input_collection by key_column and writes to output_folder
parquet_to_file(input_collection, key_column, value_columns, output_file, key_format, value_format)
read parquet of sorted input_collection and writes to output_file the key and values using format key_format and value_format
For development
Either locally, or in gitpod (do export PIP_USER=false there)
Setup a virtualenv:
python3 -m venv .env
source .env/bin/activate
pip install -e .
to run tests:
pip install -r requirements-test.txt
then
make lint
make test
You can use make black to reformat the code
python -m pytest -x -s -v tests -k "dummy" to run a specific test
Metadata
Release files for static-ondisk-kv 1.1.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| static_ondisk_kv-1.1.2.tar.gz | 5.1 kB | Without provenance | Details for static_ondisk_kv-1.1.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| static_ondisk_kv-1.1.2-py3-none-any.whl | Python 3 | none | any | Details for static_ondisk_kv-1.1.2-py3-none-any.whl |
Total release size: 12.1 kB
Release files / static_ondisk_kv-1.1.2.tar.gz
| Download URL | static_ondisk_kv-1.1.2.tar.gz |
|---|---|
| Size | 5.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0bdbe46080cf5069eaaf7a51490052076706c03f59d6bc9d489774632e29155b
|
|
BLAKE2b-256 checksum How to use checksums |
10e8764716680ee9582f8e72938f1cae752adc116db608ad6000c198e08ef6e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.0 CPython/3.8.12
|
Release files / static_ondisk_kv-1.1.2-py3-none-any.whl
| Download URL | static_ondisk_kv-1.1.2-py3-none-any.whl |
|---|---|
| Size | 7.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0a9bfe32a1a284aa93e84177addd3de211ff12185dffeb66560b39db98778dd2
|
|
BLAKE2b-256 checksum How to use checksums |
d441b08bcfcc728aef0475a886f0eb1de1675e12223c4a12c9be428e3f57dcf4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.0 CPython/3.8.12
|
Security for static-ondisk-kv 1.1.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.