deltalake-redis-lock
Project description
deltalake-redis-lock
A library creating an interface for a write lock for delta-rs.
Library Usage
When using this client, it can be used from multiple hosts. Below follow a minimal example to mimic this behaviour.
Redis Env Variables
Make sure to set these envs
before executing code.
REDIS_HOST=<host>
REDIS_PORT=<port> # default 6739
REDIS_DB=<0> # default 0
Concurrent Write Example
# run.py
import logging
import os
import string
from random import choices
from multiprocessing import Pool
from pandas import DataFrame
from src.delta_rs import write_redis_lock_deltalake
def fake_worker(args):
df, table_name = args
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s [%(levelname)s] %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
write_redis_lock_deltalake(
table_or_uri=f"{os.getcwd()}/{table_name}",
lock_table_name=table_name,
mode="append",
data=df,
overwrite_schema=True,
)
def define_datasets(_table_name: str) -> None:
df1 = DataFrame({"id": [1]})
df2 = DataFrame({"id": [2]})
df3 = DataFrame({"id": [3]})
df4 = DataFrame({"id": [4]})
datasets = [(df1, table_name), (df2, table_name), (df3, table_name), (df4, table_name)]
with Pool() as pool:
pool.map(fake_worker, datasets)
def generate_random_string(length):
return "".join(choices(string.ascii_lowercase, k=length))
if __name__ == '__main__':
random_string = generate_random_string(3)
table_name = f"test_run_{random_string}"
define_datasets(_table_name=table_name)
This can be exeucted with something like:
seq 2 | xargs -I{} -P 2 poetry run python run.py
Setup From Scratch
Requirement
- ^python3.9
- poetry 1.1.13
- make (GNU Make 3.81)
Setup
make setup-environment
Update package
make update
Test
export PYTHONPATH="${PYTHONPATH}:src"
make test type=unit
Docker
The reason docker
is used in the source code here, is to be able to build up an encapsulated
environment of the codebase, and do unit/integration and load tests
.
make build-container-image
make get-container-info-environment
make run-container-tests type=unit
Docker
make build-container-image DOCKER_BUILD="buildx build --platform linux/amd64" CONTEXT=.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for deltalake-redis-lock-0.0.1a1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 307960fb9e60722cb4d4a31222504637f23b72cf39fd59dd5d470d0438139cab |
|
MD5 | fd28921d6774cb832b11ca57b47f4858 |
|
BLAKE2b-256 | 59308c5014c06ebec466aba0a55831c74f76c6ee422e355d8b3817fc2e6c63bc |
Close
Hashes for deltalake_redis_lock-0.0.1a1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c849d52a3e8d5182a87dea47e15a5a4347fb809dfd06fb3da2da044867c54f9e |
|
MD5 | d4c2a257225e278cd7ddab61b3c6f2eb |
|
BLAKE2b-256 | 24e33444f25cbf59ec98f2a0e8f040dd7f7cf9e25f0dd3a24f610cc8eeff10f9 |