MutableMapping interfaces for common cloud storage providers
Project description
cloud-mappings
MutableMapping implementations for common cloud storage providers
For now Azure Blob Storage, Google Cloud Storage, and AWS S3 are implemented. Contributions of new providers are welcome.
Installation
with pip:
pip install cloud-mappings
By default, cloud-mappings
doesn't install any of the required storage providers dependencies. If you would like to install them alongside cloud-mappings
you may run any combination of:
pip install cloud-mappings[azure,gcp,aws]
Instantiation
AzureBlobMapping:
from cloudmappings import AzureBlobMapping
cm = AzureBlobMapping.with_pickle(
account_url="AZURE_BLOB_STORAGE_URL",
container_name="CONTAINER_NAME",
credential=AZURE_CREDENTIAL_OBJECT,
)
GoogleCloudStorageMapping:
from cloudmappings import GoogleCloudStorageMapping
cm = GoogleCloudStorageMapping.with_pickle(
project="GCP_PROJECT",
credentials=GCP_CREDENTIALS_OBJECT,
bucket_name="BUCKET_NAME",
)
AWSS3Mapping:
from cloudmappings import AWSS3Mapping
cm = AWSS3Mapping.with_pickle(
bucket_name="AWS_BUCKET_NAME",
silence_warning=False,
)
Note that AWS S3 does not support server-side atomic requests, so it is not recommended for concurrent use. A warning is printed out by default but may be silenced by passing silence_warning=True
.
Usage
Use it just like a standard dict()
!
cm["key"] = 1000
cm["key"] # returns 1000
del cm["key"]
"key" in cm # returns false
Cloud Sync
Each cloud-mapping
keeps an internal dict of etags which it uses to ensure it is only reading/overwriting/deleting data it expects to. If the value in storage is not what the cloud-mapping
expects, a cloudmappings.errors.KeySyncError()
will be thrown. If you know what you are doing and want your operation to go through anyway, you will need to sync your cloud-mapping
with the cloud by calling either .sync_with_cloud()
to sync all keys or .sync_with_cloud(key)
to sync a specific key. By default .sync_with_cloud()
is called on instantiation of a cloud-mapping
if the underlying provider storage already exists. You may skip this initial sync by passing an additional sync_initially=False
parameter when you instantiate your cloud-mapping
.
Serialisation
If you don't call .with_pickle()
and instead pass your providers configuration directly to the cloud-mapping
class, you will get a "raw" cloud-mapping
which only accepts byte-likes as values. You may build your own serialisation either using zict; or by calling .with_buffers([dumps_1, loads_1, dumps_2, loads_2, ...])
where dumps
and loads
are the ordered functions to serialise and deserialise your data respectively.
The following exist as common starting points: .with_pickle()
, .with_json()
, .with_json_zlib()
.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cloud_mappings-0.7.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e5381d33f49efc823379a17ce4f41f7b1b5ac4279419816cfe3e2ec2655696a6 |
|
MD5 | 127b4edf09c9eaf52e07861e29ce2e7b |
|
BLAKE2b-256 | 6810f0531628247ecc310da321ef3511a6415e13f9a82170840646e37b35b8d8 |