Skip to main content

Amazon S3 filesystem for PyFilesystem2, with streaming and non-strict by default

Project description

S3FS is a PyFilesystem interface to Amazon S3 cloud storage.

As a PyFilesystem concrete class, S3FS allows you to work with S3 in the same way as any other supported filesystem.

Installing

You can install S3FS from pip as follows:

pip install fs-s3fs-ng

Opening a S3FS

Open an S3FS by explicitly using the constructor:

from fs_s3fs import S3FS
s3fs = S3FS('mybucket')

Or with a FS URL:

from fs import open_fs
s3fs = open_fs('s3://mybucket')

Downloading Files

To download files from an S3 bucket, open a file on the S3 filesystem for reading, then write the data to a file on the local filesystem. Here’s an example that copies a file example.mov from S3 to your HD:

from fs.tools import copy_file_data
with s3fs.open('example.mov', 'rb') as remote_file:
    with open('example.mov', 'wb') as local_file:
        copy_file_data(remote_file, local_file)

Although it is preferable to use the higher-level functionality in the fs.copy module. Here’s an example:

from fs.copy import copy_file
copy_file(s3fs, 'example.mov', './', 'example.mov')

Uploading Files

You can upload files in the same way. Simply copy a file from a source filesystem to the S3 filesystem. See Moving and Copying for more information.

ExtraArgs

S3 objects have additional properties, beyond a traditional filesystem. These options can be set using the upload_args and download_args properties. which are handed to upload and download methods, as appropriate, for the lifetime of the filesystem instance.

For example, to set the cache-control header of all objects uploaded to a bucket:

import fs, fs.mirror
s3fs = S3FS('example', upload_args={"CacheControl": "max-age=2592000", "ACL": "public-read"})
fs.mirror.mirror('/path/to/mirror', s3fs)

see the Boto3 docs for more information.

acl and cache_control are exposed explicitly for convenience, and can be used in URLs. It is important to URL-Escape the cache_control value in a URL, as it may contain special characters.

import fs, fs.mirror
with open fs.open_fs('s3://example?acl=public-read&cache_control=max-age%3D2592000%2Cpublic') as s3fs
    fs.mirror.mirror('/path/to/mirror', s3fs)

S3 URLs

You can get a public URL to a file on a S3 bucket as follows:

movie_url = s3fs.geturl('example.mov')

Documentation

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fs-s3fs-ng-1.1.2b4.tar.gz (10.8 kB view details)

Uploaded Source

Built Distribution

fs_s3fs_ng-1.1.2b4-py2.py3-none-any.whl (12.5 kB view details)

Uploaded Python 2Python 3

File details

Details for the file fs-s3fs-ng-1.1.2b4.tar.gz.

File metadata

  • Download URL: fs-s3fs-ng-1.1.2b4.tar.gz
  • Upload date:
  • Size: 10.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0 requests-toolbelt/0.9.1 tqdm/4.38.0 CPython/3.6.8

File hashes

Hashes for fs-s3fs-ng-1.1.2b4.tar.gz
Algorithm Hash digest
SHA256 d6b7d789fed591f33d9d33c1c329c0ea38b0b80401aa7877170a176f0e7bc03d
MD5 fcebdb6d53e408775b5bf2c8a485c29a
BLAKE2b-256 87727afbfd2a9f51400495fbe30b9aaddb98596fe6bc3beb5b491d61666db470

See more details on using hashes here.

File details

Details for the file fs_s3fs_ng-1.1.2b4-py2.py3-none-any.whl.

File metadata

  • Download URL: fs_s3fs_ng-1.1.2b4-py2.py3-none-any.whl
  • Upload date:
  • Size: 12.5 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0 requests-toolbelt/0.9.1 tqdm/4.38.0 CPython/3.6.8

File hashes

Hashes for fs_s3fs_ng-1.1.2b4-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 49a013341193a7fe842cfce10b1f1cc2669bb239014a9c1fed632de0a6a631da
MD5 91ae8fd380254e5f98204057262db0a4
BLAKE2b-256 19b937fb05b97d14643b7b9df94e1bfc23070a964b6491fea32f6aab291ec59d

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page