metabaiji 2.12.0

High-level Python abstraction layer for Amazon S3

This is an active fork of baiji, a high-level Python abstraction layer for Amazon S3:

1. An `open <https://docs.python.org/2/library/functions.html#open>`__-like context handler which allows using S3 keys and local files interchangeably.

   • When reading S3, contents are first written to a temporary local file.

   • When writing S3, contents are written to a temporary local file, and uploaded on close.

2. An s3 CLI for listing, copying, syncing, and other common activities.

The fork’s goals are modest:

• Keep the library working in current versions of Python and other tools.

• Make bug fixes.

• Provide API stability and backward compatibility with the upstream version.

• Respond to community contributions.

It’s used by related forks such as lace.

Features

• Works without an S3 connection (with local files).

• Supports multiprocess parallelism for copying lots of files.

• Supports Python 2.7 and uses boto2.

• Supports OS X, Linux, and Windows.

• Tested and production-hardened.

Examples

with s3.open('s3://example/info.txt', 'w') as f:
    f.write('hello')

with s3.open('file:///home/me/info.txt', 'w') as f:
    f.write('hello')

with s3.open('s3://example/info.txt', 'r') as f:
    contents = f.readlines()

with s3.open('file:///home/me/info.txt', 'r') as f:
    contents = f.readlines()

s3 cp foo.txt s3://example/bar.txt
s3 cp s3://example/bar.txt s3://another-example/bazinga.txt
s3 rm s3://example/bar.txt

Development

pip install -r requirements_dev.txt
rake test
rake lint

TODO

1. Migrate credentials to ~/.aws/credentials or env, and deprecate AWS credential support in ~/.bodylabs.

2. Move baiji.util.parallel into a separate library.

3. Upgrade to boto3.

Contribute

• Issue Tracker: github.com/bodylabs/baiji/issues

• Source Code: github.com/bodylabs/baiji

Pull requests welcome!

Support

If you are having issues, please let us know.

License

The project is licensed under the Apache license, version 2.0.