No project description provided
Project description
REDAST - Remote Data Storage
Redast provides a flexible interface for storing data in different types of storages. The organization of data storage is carried out according to the principle of a data lake.
Installation
pip install redast
Or from GitHub
git clone git@github.com:rilshok/redast.git
cd redast
pip install -e .
or
pip install git+https://github.com/rilshok/redast.git
Supported storage methods
Local storage methods
Memory storage
Local storage of data in RAM
from redast import Storage, Memory
storage = Storage(Memory())
Drive storage
Local storage of data on drive. To open a storage, you must specify the root
folder.
This call will create an empty directory ./myStorage
.
from redast import Storage, Drive
drive = Drive(root="myStorage", create=True)
storage = Storage(drive)
Cloud storage methods
Mega
MEGA is a cloud storage and file hosting service offered by MEGA Limited.
from redast import Storage, MegaCloud
email = input()
password = input()
mega = MegaCloud(email=email, password=password, root='mystorage')
storage = Storage(mega)
Data storage interface
Push data into storage
key = storage.push(b"hello world")
print(key)
021ced8799296ceca557832ab941a50b4a11f83478cf141f51f933f653ab9fbcc05a037cddbed06e309bf334942c4e58cdf1a46e237911ccd7fcf9787cbc7fd0
Pull data from storage
data = storage.load(key)
print(data)
b'hello world'
Check for data in storage
key = storage.push(b"hello world")
ok = storage.exists(key)
bad = storage.exists("brokenkey")
print(ok, bad)
True False
Pop data from storage
key = storage.push(b"hello world")
data = storage.pop(key)
exist = storage.exists(key)
print(data, exist)
b'hello world' False
Custom links
The link
method links the saved data to the user identifier.
storage.link("hello").push(b"hello world")
data = storage.link("hello").load()
print(data)
b'hello world'
Any python object can act as an identifier, even a lambda function.
key = lambda x: x**2
storage.link(key).push(b"squaring")
qrt = storage.link(key).load()
print(qrt)
b'squaring'
A sequence of objects can be used as an identifier.
storage.link("hello", "world").push(b"hello world")
data = storage.link("hello", "world").pop()
print(data)
b'hello world'
Data packaging
Some form of pre-processing may be required to store the data. For large storages, data compression may be appropriate. Public storage may require encryption. The REDAST supports different preprocessing.
Compression
data = b"hello"*5
key = storage.compression.push(data)
compressed = storage.load(key)
data = storage.compression.load(key)
print(len(compressed), len(data), compressed, data, sep="\n")
16
25
b'x\x9c\xcbH\xcd\xc9\xc9\xcf\xc0B\x00\x00\x86\xc4\ne'
b'hellohellohellohellohello'
Base64
key = storage.base64.push(b"hello world")
data_base64 = storage.load(key)
data = storage.base64.load(key)
print(data_base64, data, sep="\n")
b'aGVsbG8gd29ybGQ='
b'hello world'
Pickling of python objects
Different types of pre-processing can be combined into a single conveyor line
data = dict(a=1, b=2)
key = storage.pickling.push(data)
data = storage.pickling.load(key)
print(data)
{'a': 1, 'b': 2}
Encryption
Encryption requires an encryption key or user password, from which the encryption key will be generated. When using encryption with a password, you must choose an arbitrary seed to create the salt.
You can generate a random key
from redast.core import Encryption
secret = Encryption.generate_key()
print(secret)
pcu398oGPbEIoc8UgXGnmjJUZOId1bQRWJ5VmpH2gQo=
Generating an encryption key using a passphrase
from redast.core import Encryption
secret1 = Encryption.generate_key(password="mypassword", seed=2022)
secret2 = Encryption.generate_key(password="mypassword", seed=2022)
secret3 = Encryption.generate_key(password="mypassword", seed=2021)
print(secret1, secret2, secret3, sep="\n")
DZ6xJLl9D7MkHkYt6JVef1j6o36KU7XzXzYRZrUDt0w=
DZ6xJLl9D7MkHkYt6JVef1j6o36KU7XzXzYRZrUDt0w=
I4sopjxisx6wb7dCkOtwoBxDvo5lnb6tctcpdpS5jGg=
Encryption with an encryption key
from redast.core import Encryption
secret = Encryption.generate_key()
key = storage.encryption(key=secret).push(b"topsecret")
encrypted = storage.load(key)
decrypted = storage.encryption(key=secret).load(key)
print(encrypted, decrypted, sep="\n")
b"\x00k\xd0=\x81\xedh\x94x\xed\xb0\x14\xe3\r\x96'"
b'topsecret'
Encryption with a password
key = storage.encryption(password="mypassword", seed=10).push(b"topsecret")
encrypted = storage.load(key)
decrypted = storage.encryption(password="mypassword", seed=10).load(key)
print(encrypted, decrypted, sep="\n")
b'\xe4\xc6\x0bc\xd0\x92\xcb\xaeQ\x0ey&\x83\xb9\x9d@'
b'topsecret'
Pipeline data packing
In this example, the data will be converted in the following order before being stored:
- First, the object to be saved will be pickled
- After pickling, the result will be compressed
- The compression result is then encrypted
- The result of encryption is converted to base64
- The base 64 string is pushed into storage
data = [("hello", "world"), "foo"]
auth = dict(
password="mypassword",
seed=2022,
)
pipe = storage.base64.encryption(**auth).compression(level=9).pickling
key = pipe.push(data)
stored = storage.load(key)
data = pipe.load(key)
print(stored, data, sep="\n")
b'z2bqXGPZoXkN_-3LB9ocrBeWi4rax7le4wJncKe1Cnz7ep-KoHgFDeV2DisMq6Az'
[('hello', 'world'), 'foo']
The default packing options for pipeline data can be set when creating the storage itself
from redast import Storage, Memory
data = [("hello", "world"), "foo"]
storage = Storage(
Memory(),
compression = 9,
encryption_password = "mypassword",
encryption_seed = 2022,
)
pipe = storage.base64.encryption.compression.pickling
key = pipe.push(data)
stored = storage.load(key)
data = pipe.load(key)
print(stored, data, sep="\n")
b'z2bqXGPZoXkN_-3LB9ocrBeWi4rax7le4wJncKe1Cnz7ep-KoHgFDeV2DisMq6Az'
[('hello', 'world'), 'foo']
Pipeline data packing with custom links
from redast import Storage, Memory
storage = Storage(Memory())
pipe = storage.base64.compression
data = b"hello world"
pipe.link("mylink").push(data)
data = pipe.link("mylink").load()
print(data)
b'hello world'
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.