MutableMapping interfaces for common cloud storage providers
Project description
cloud-mappings
MutableMapping implementations for common cloud storage providers
For now Azure Blob Storage, Google Cloud Storage, and AWS S3 are implemented. Contributions of new providers are welcome.
Installation
with pip:
pip install cloud-mappings
By default, cloud-mappings
doesn't install any of the required storage providers dependencies. If you would like to install them alongside cloud-mappings
you may run any combination of:
pip install cloud-mappings[azure,gcp,aws]
Instantiation
AzureBlobMapping:
from cloudmappings import AzureBlobMapping
cm = AzureBlobMapping.with_pickle(
account_url="AZURE_BLOB_STORAGE_URL",
container_name="CONTAINER_NAME",
credential=AZURE_CREDENTIAL_OBJECT,
)
GoogleCloudStorageMapping:
from cloudmappings import GoogleCloudStorageMapping
cm = GoogleCloudStorageMapping.with_pickle(
project="GCP_PROJECT",
credentials=GCP_CREDENTIALS_OBJECT,
bucket_name="BUCKET_NAME",
)
AWSS3Mapping:
from cloudmappings import AWSS3Mapping
cm = AWSS3Mapping.with_pickle(
bucket_name="AWS_BUCKET_NAME",
silence_warning=False,
)
Note that AWS S3 does not support server-side atomic requests, so it is not recommended for concurrent use. A warning is printed out by default but may be silenced by passing silence_warning=True
.
Usage
Use it just like a standard dict()
!
cm["key"] = 1000
cm["key"] # returns 1000
del cm["key"]
"key" in cm # returns false
Cloud Sync
Each cloud-mapping
keeps an internal dict of etags which it uses to ensure it is only reading/overwriting/deleting data it expects to. If the value in storage is not what the cloud-mapping
expects, a cloudmappings.errors.KeySyncError()
will be thrown. If you know what you are doing and want your operation to go through anyway, you will need to sync your cloud-mapping
with the cloud by calling either .sync_with_cloud()
to sync all keys or .sync_with_cloud(key)
to sync a specific key. By default .sync_with_cloud()
is called on instantiation of a cloud-mapping
if the underlying provider storage already exists. You may skip this initial sync by passing an additional sync_initially=False
parameter when you instantiate your cloud-mapping
.
Serialisation
If you don't call .with_pickle()
and instead pass your providers configuration directly to the CloudMapping
class, you will get a "raw" cloud-mapping
which accepts only byte-likes as values. Along with the .with_pickle()
serialisation utility, .with_json()
and .with_json_zlib()
also exist.
You may build your own serialisation either using zict; or by calling .with_buffers([dumps_1, dumps_2, ..., dumps_N], [loads_1, loads_2, ..., loads_N])
, where dumps
and loads
are the ordered functions to serialise and parse your data respectively.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cloud_mappings-0.7.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1e9cd1e6f606d13771ba26e669abc02f485a727181e909b48bc3172c9e29dbe4 |
|
MD5 | dd0f0639e32f1f699c1bdb3f57846a86 |
|
BLAKE2b-256 | 022bd41369187c5ff374d3b713f44c80641f71fde2ee259942f413679aaa287b |