Skip to main content

convenience functions for serialization to s3

Project description

Shalosh -- convenience functions for serialization to s3

@author Boris Gorelik boris@gorelik.net @licencse This module is distributed under the MIT license

What's this?

In many cases, I had to implement an object that needs to save stuff either to the local fillesystem, or, depending on configuration, to a S3 instance. The provided module provides a single object that does exactly that: you initialize the object once, and then, you can use the various functions that it provides such as:

  • ls
  • path_exists
  • rm
  • rmtree
  • ls
  • load_pickle, dump_pickle
  • load_json, dump_json

Testing & Usage

The packcage provides a set of unit tests that are located in the tests directory. Read through these files to learn how the module is used. In order to be able to run the tests, you have to create a secret folder in the directory of this README.md. Create a confi.json file in that folder that looks like this:

{
  "s3": {
    "defaultBucket": "bucket",
    "accessKey": "ABCDEFGHIJKLMNOP",
    "accessSecret": "/accessSecretThatOnlyYouKnow"
  }
}

The following is a short example of how to use sshalosh. In that example, we decide whether we want to work with the local filesystem or with S3, create a serizlizer object according to this decision, and then work as usual. The actual code remains the same

if work_with_s3:
    s3_config = {
      "s3": {
        "defaultBucket": "bucket",
        "accessKey": "ABCDEFGHIJKLMNOP",
        "accessSecret": "/accessSecretThatOnlyYouKnow"
      }
    }

else:
    s3_config = None
serializer = sshalosh.Serializer(s3_config)

# Done! From now on, you only need to deal with the business logic, not the house-keeping

# Load data & model
data = serializer.load_json('data.json')
model = serializer.load_pickle('model.pkl')

# Update
data = update_with_new_examples()
model.fit(data)

# Save updated objects
serializer.dump_json(data, 'data.json')
serializer.dump_pickle(model, 'model.pkl')

As simple as that!

What does the name mean?

In Hebrew, "shalosh" means "three". Thus, s-shalosh is s3.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release. See tutorial on generating distribution archives.

Built Distribution

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page