Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Utility class in Python for finding, saving, and deleting files that are either on Amazon S3, or on the local filesystem.

Project Description

Utility class in Python for finding, saving, and deleting files that are either on Amazon S3, or on the local filesystem.

The idea behind this, is that you can pass exactly the same data (in particular, the same file path) irrespective of whether you’re working with a local or S3-based file. All you have to do, on each call, is to specify ‘s3’ for S3, or None for local.

This is great for apps where all files are stored on the local filesystem in one environment (e.g. development), and on S3 in another environment (e.g. production). The only thing you have to work out at run-time, is whether to specify ‘s3’ or None - the rest of your file-handling code stays the same.

Example

For a complete, working Flask app that demonstrates s3-saver in action, have a look at flask-s3-save-example.

Usage

from io import BytesIO
import os
from s3_saver import S3Saver

bucket_name = 's3-test-foobar-whizbang'

# Absolute path to your project's root
static_root_parent = '/home/mrfoo/pyprojectfoo'

# Absolute path to dir where file gets saved
base_path = os.path.join(static_root_parent, 'static/uploads')

# For a dummy object that attributes can get saved to.
# In most real-life cases, this would be an ORM model
# (e.g. inherited from django.db.models.Model for Django, or
# sqlalchemy.ext.declarative.declarative_base() for SQLAlchemy).
class Thingy(object):
    pass

# Load a sample file into a temp object
filename = 'whizbang.jpg'
filepath = os.path.join(base_path, filename)
thingy = Thingy()
temp_file = BytesIO()

# Read a local file into a BytesIO object.
# For most real-life cases, you'd instead load a file uploaded
# in a HTTP POST request into a BytesIO. E.g. in a Flask route:
# from flask import request
# request.files['thingy_image'].save(temp_file)
f = open('/home/mrfoo/photos/hobbies/whizbang.jpg', 'rb')
f.seek(0)
one_mb = 1024*1024

t = f.read(one_mb)
while t:
    temp_file.write(t)
    t = f.read(one_mb)

f.close()

# Initialize the saver
image_saver = S3Saver(
    storage_type='s3',
    bucket_name=bucket_name,
    access_key_id='XXXXX',
    access_key_secret='YYYYY',
    field_name='image',
    storage_type_field='image_storage_type',
    bucket_name_field='image_storage_bucket_name',
    filesize_field='image_filesize',
    base_path=base_path,
    static_root_parent=static_root_parent)

# Save the file to S3, in the specified bucket,
# at 'static/uploads/whizbang.jpg'
image_saver.save(temp_file, filename, thingy)

print(thingy.image) # 'whizbang.jpg'
print(thingy.image_storage_type) # 's3'
print(thingy.image_storage_bucket_name) # 's3-test-foobar-whizbang'

# In most real-life cases, you'd persist the 'thingy' object
# at this point. E.g. in SQLAlchemy:
# db.session.add(thingy)
# db.session.commit()

# Now save the file locally,
# at '/home/mrfoo/pyprojectfoo/static/uploads/whizbang.jpg'
image_saver.storage_type = None
image_saver.save(temp_file, filename, thingy)

print(thingy.image) # 'whizbang.jpg'
print(thingy.image_storage_type) # ''
print(thingy.image_storage_bucket_name) # ''

# Find files on S3, searching by key prefix.
# Prints:
# [u'static/uploads/whizbang.jpg']
print([k.name for k in image_saver.find_by_path(
    '/home/mrfoo/pyprojectfoo/static/uploads/whizb',
    storage_type='s3',
    bucket_name=bucket_name)])

# Find files locally, searching by glob.
# Prints:
# ['/home/mrfoo/pyprojectfoo/static/uploads/whizbang.jpg']
print([k for k in image_saver.find_by_path(
    '/home/mrfoo/pyprojectfoo/static/uploads/whizb',
    storage_type=None,
    bucket_name=bucket_name)])

# Delete the file on S3.
image_saver.delete(
    '/home/mrfoo/pyprojectfoo/static/uploads/whizbang.jpg',
    storage_type='s3',
    bucket_name=bucket_name)

# Delete the file locally.
image_saver.delete(
    '/home/mrfoo/pyprojectfoo/static/uploads/whizbang.jpg',
    storage_type=None,
    bucket_name=bucket_name)
Release History

Release History

This version
History Node

0.1.4

History Node

0.1.3

History Node

0.1.2

History Node

0.1.1

History Node

0.1.0

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
s3_saver-0.1.4-py2.py3-none-any.whl (6.6 kB) Copy SHA256 Checksum SHA256 py2.py3 Wheel Aug 28, 2015

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting