deltalake-redis-lock
Project description
deltalake-redis-lock
A library creating an interface for a write lock for delta-rs.
Library Usage
When using this client, it can be used from multiple hosts. Below follow a minimal example to mimic this behaviour.
Redis Env Variables
Make sure to set these envs
before executing code.
REDIS_HOST=<host>
REDIS_PORT=<port> # default 6739
REDIS_DB=<0> # default 0
Concurrent Write Example
# run.py
import logging
import os
import string
from random import choices
from multiprocessing import Pool
from pandas import DataFrame
from deltalake_redis_lock import write_redis_lock_deltalake
def fake_worker(args):
df, table_name = args
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s [%(levelname)s] %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
write_redis_lock_deltalake(
table_or_uri=f"{os.getcwd()}/{table_name}",
lock_table_name=table_name,
mode="append",
data=df,
overwrite_schema=True,
)
def define_datasets(_table_name: str) -> None:
df1 = DataFrame({"id": [1]})
df2 = DataFrame({"id": [2]})
df3 = DataFrame({"id": [3]})
df4 = DataFrame({"id": [4]})
datasets = [(df1, table_name), (df2, table_name), (df3, table_name), (df4, table_name)]
with Pool() as pool:
pool.map(fake_worker, datasets)
def generate_random_string(length):
return "".join(choices(string.ascii_lowercase, k=length))
if __name__ == '__main__':
random_string = generate_random_string(3)
table_name = f"test_run_{random_string}"
define_datasets(_table_name=table_name)
This can be exeucted with something like:
seq 2 | xargs -I{} -P 2 poetry run python run.py
Setup From Scratch
Requirement
- ^python3.9
- poetry 1.1.13
- make (GNU Make 3.81)
Setup
make setup-environment
Update package
make update
Test
export PYTHONPATH="${PYTHONPATH}:src"
make test type=unit
Docker
The reason docker
is used in the source code here, is to be able to build up an encapsulated
environment of the codebase, and do unit/integration and load tests
.
make build-container-image DOCKER_BUILD="buildx build --platform linux/amd64" CONTEXT=.
make run-container-tests type=unit
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for deltalake-redis-lock-0.0.1a6.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | d43b6e52e56911b2ef39d42b9036504beb6872617626a48a6e9867ca5d07a4be |
|
MD5 | ba99550735c21a624c53b6ea5c23bc20 |
|
BLAKE2b-256 | 23eb9c79583777706bbbb4816fea25534b2f7aafcbcaf035b46ebc2033474b47 |
Close
Hashes for deltalake_redis_lock-0.0.1a6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d071e421cc7b16d8db896bef191ef46346b96a0c704ae7a4bcf1757a8c7d0cd5 |
|
MD5 | f8161ad6c598ceb5110dbe3318bd7df1 |
|
BLAKE2b-256 | 2df2c3abe9e542262925fb224b2837c8eb2ac514958e909689f40e77da1403a3 |