deltalake-redis-lock
Project description
deltalake-redis-lock
A library creating an interface for a write lock for delta-rs.
Library Usage
When using this client, it can be used from multiple hosts. Below follow a minimal example to mimic this behaviour.
Redis Env Variables
Make sure to set these envs
before executing code.
REDIS_HOST=<host>
REDIS_PORT=<port> # default 6739
REDIS_DB=<0> # default 0
Concurrent Write Example
# run.py
import logging
import os
import string
from random import choices
from multiprocessing import Pool
from pandas import DataFrame
from deltalake_redis_lock import write_redis_lock_deltalake
def fake_worker(args):
df, table_name = args
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s [%(levelname)s] %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
write_redis_lock_deltalake(
table_or_uri=f"{os.getcwd()}/{table_name}",
lock_table_name=table_name,
mode="append",
data=df,
overwrite_schema=True,
)
def define_datasets(_table_name: str) -> None:
df1 = DataFrame({"id": [1]})
df2 = DataFrame({"id": [2]})
df3 = DataFrame({"id": [3]})
df4 = DataFrame({"id": [4]})
datasets = [(df1, table_name), (df2, table_name), (df3, table_name), (df4, table_name)]
with Pool() as pool:
pool.map(fake_worker, datasets)
def generate_random_string(length):
return "".join(choices(string.ascii_lowercase, k=length))
if __name__ == '__main__':
random_string = generate_random_string(3)
table_name = f"test_run_{random_string}"
define_datasets(_table_name=table_name)
This can be exeucted with something like:
seq 2 | xargs -I{} -P 2 poetry run python run.py
Setup From Scratch
Requirement
- ^python3.9
- poetry 1.1.13
- make (GNU Make 3.81)
Setup
make setup-environment
Update package
make update
Test
export PYTHONPATH="${PYTHONPATH}:src"
make test type=unit
Docker
The reason docker
is used in the source code here, is to be able to build up an encapsulated
environment of the codebase, and do unit/integration and load tests
.
make build-container-image DOCKER_BUILD="buildx build --platform linux/amd64" CONTEXT=.
make get-container-info-environment
make run-container-tests type=unit
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for deltalake-redis-lock-0.0.1a4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8bffa7bba17a697d58781e4ab41ec27560fb47641c00b052ac52c47683a481d0 |
|
MD5 | 5183148402977ce19f3c7f4530ba5bf7 |
|
BLAKE2b-256 | 5db9a25329134ad2268d9fbf9ba303db17fd164157c3605c6b90957f48c2e22a |
Close
Hashes for deltalake_redis_lock-0.0.1a4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1da1d50b65cdd6b2c2108934d3899d1507f7c40bc87a1a79f1a9a3742e87828c |
|
MD5 | 48ad09916ffec60506eb6dff87e91001 |
|
BLAKE2b-256 | 2fdf6e8820d4724c5308b207633031dd8e665aa1464fbb2d7c312a1936e7b7a3 |