Skip to main content

convenience functions for serialization to s3

Project description

Shalosh -- convenience functions for serialization to s3

@author Boris Gorelik boris@gorelik.net @licencse This module is distributed under the MIT license

What's this?

In many cases, I had to implement an object that needs to save stuff either to the local fillesystem, or, depending on configuration, to a S3 instance. The provided module provides a single object that does exactly that: you initialize the object once, and then, you can use the various functions that it provides such as:

  • ls
  • path_exists
  • rm
  • rmtree
  • ls
  • load_pickle, dump_pickle
  • load_json, dump_json

Testing & Usage

The packcage provides a set of unit tests that are located in the tests directory. Read through these files to learn how the module is used. In order to be able to run the tests, you have to create a secret folder in the directory of this README.md. Create a confi.json file in that folder that looks like this:

{
  "s3": {
    "defaultBucket": "bucket",
    "accessKey": "ABCDEFGHIJKLMNOP",
    "accessSecret": "/accessSecretThatOnlyYouKnow"
  }
}

The following is a short example of how to use sshalosh. In that example, we decide whether we want to work with the local filesystem or with S3, create a serizlizer object according to this decision, and then work as usual. The actual code remains the same

if work_with_s3:
    s3_config = {
      "s3": {
        "defaultBucket": "bucket",
        "accessKey": "ABCDEFGHIJKLMNOP",
        "accessSecret": "/accessSecretThatOnlyYouKnow"
      }
    }

else:
    s3_config = None
serializer = sshalosh.Serializer(s3_config)

# Done! From now on, you only need to deal with the business logic, not the house-keeping

# Load data & model
data = serializer.load_json('data.json')
model = serializer.load_pickle('model.pkl')

# Update
data = update_with_new_examples()
model.fit(data)

# Save updated objects
serializer.dump_json(data, 'data.json')
serializer.dump_pickle(model, 'model.pkl')

As simple as that!

What does the name mean?

In Hebrew, "shalosh" means "three". Thus, s-shalosh is s3.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distribution

sshalosh_borisgorelik-0.0.3-py3-none-any.whl (4.4 kB view details)

Uploaded Python 3

File details

Details for the file sshalosh_borisgorelik-0.0.3-py3-none-any.whl.

File metadata

  • Download URL: sshalosh_borisgorelik-0.0.3-py3-none-any.whl
  • Upload date:
  • Size: 4.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.6.1 requests/2.24.0 setuptools/50.3.1.post20201107 requests-toolbelt/0.9.1 tqdm/4.50.2 CPython/3.8.5

File hashes

Hashes for sshalosh_borisgorelik-0.0.3-py3-none-any.whl
Algorithm Hash digest
SHA256 6413322edc92fd18d6cd9db24694de36da36f510f4cb1683cf9f7b566134c36a
MD5 f11d8c5fc9dab44fe5a15056bc486201
BLAKE2b-256 5b7530e19ee45aa59edc818c995cecf5c215df96ac528912f9065c4f247ef084

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page