Skip to main content

S3 Object Store. Store and retrieve python objects in s3 simply..

Project description

S3 Object Store

badge

Simple pythonic wrapper to use s3 as a python object store.

Provides simple object storage methods; and a Dict-like interface for more complicated uses.

Examples

Basic usage:

from s3os import store_simple, retrieve_simple, delete_simple

my_object = [1, 2, 3]

store_simple("my_key", my_object)

assert retrieve_simple("my_key") == my_object

delete_simple("my_key")

The above example uses a global namespace in the bucket "s3os" - i.e. all the default settings of this package.

You can specify your own namespaces (i.e. buckets) as follows:

from s3os import store, retrieve, delete, ObjectLocation, Bucket

my_bucket = Bucket("my_bucket")
my_object_location = ObjectLocation("my_key", bucket=my_bucket)
my_object = [1, 2, 3]

store(my_object_location, my_object)

assert retrieve(my_object_location) == my_object

delete(my_object_location)

Or simply use s3 like a normal python dictionary:

from s3os import S3Dict, S3DictConfig, Bucket

my_bucket = Bucket("my_bucket")
s3dict = S3Dict(_config=S3DictConfig(id="my_dict_id", bucket=my_bucket))

# Store information in s3
s3dict["apples"] = 5
s3dict["bananas"] = 2

...

# Later, or in a different python executable, access the same dictionary again:
my_bucket = Bucket("my_bucket")
s3dict = S3Dict(_config=S3DictConfig(id="my_dict_id", bucket=my_bucket))

print(s3dict["apples])  # 5
print(s3dict.get_all_from_s3())  # {"apples": 5, "bananas": 2}

By default, S3Dict uses an internal cache to speed up item retrieval. Set and Delete operations are always performed synchronously.

Installation

Install the package:

pip install s3os

Setup your AWS credentials. For example set these environment variables:

AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

Also see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html for more authentication options.

Note: boto.client() and Session authentication methods are not currently supported - raise an issue or submit a PR if you want them!

Development installation

Install poetry - see https://pypi.org/project/poetry/

The following command should be used to install the dependencies:

poetry install

Testing

The following command should be used to run the tests:

poetry run pytest tests

Valid AWS authentication credentials are required to run some of the tests. See setup instructions.

The tests make a very small number of calls to S3, so the cost of running the tests is negligible.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

s3os-1.0.0.tar.gz (7.7 kB view details)

Uploaded Source

Built Distribution

s3os-1.0.0-py3-none-any.whl (8.4 kB view details)

Uploaded Python 3

File details

Details for the file s3os-1.0.0.tar.gz.

File metadata

  • Download URL: s3os-1.0.0.tar.gz
  • Upload date:
  • Size: 7.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.0.0 CPython/3.8.0 Linux/5.0.0-1027-azure

File hashes

Hashes for s3os-1.0.0.tar.gz
Algorithm Hash digest
SHA256 9970b1f9e5997b175daa59936b597cffcd4100026bfc4c670c64c041787f9a59
MD5 dc7c48e07da8b04435458228fa1157ce
BLAKE2b-256 08ee3cbcab2599caf285fc62149173069cd7d95ae7d118c9c9ae3f8aa68f4c31

See more details on using hashes here.

File details

Details for the file s3os-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: s3os-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 8.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.0.0 CPython/3.8.0 Linux/5.0.0-1027-azure

File hashes

Hashes for s3os-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 2f7499fb7d1c1d6a63a2e6468d9ef5ed6ee8e0ee9520317ed101e8d5339b4693
MD5 4511fc18e66968ff35ce93d2cb9e9979
BLAKE2b-256 eb572c5bb2cf96f6f01ddd8212b8e8ff4d7c5cc47178f1375ce14ff3b34ece23

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page