Skip to main content

MinIO S3 filesystem for PyFilesystem2

Project description

MINIOFS is a PyFilesystem interface to Amazon S3 cloud storage.

As a PyFilesystem concrete class, MINIOFS allows you to work with S3 in the same way as any other supported filesystem.

Installing

You can install MINIOFS from pip as follows:

pip install fs-miniofs

Opening a MINIOFS

Open an MINIOFS by explicitly using the constructor:

from fs_miniofs import MINIOFS
miniofs = MINIOFS('mybucket')

Or with a FS URL:

from fs import open_fs
miniofs = open_fs('minio://mybucket')

Downloading Files

To download files from an S3 bucket, open a file on the S3 filesystem for reading, then write the data to a file on the local filesystem. Here’s an example that copies a file example.mov from S3 to your HD:

from fs.tools import copy_file_data
with miniofs.open('example.mov', 'rb') as remote_file:
    with open('example.mov', 'wb') as local_file:
        copy_file_data(remote_file, local_file)

Although it is preferable to use the higher-level functionality in the fs.copy module. Here’s an example:

from fs.copy import copy_file
copy_file(miniofs, 'example.mov', './', 'example.mov')

Uploading Files

You can upload files in the same way. Simply copy a file from a source filesystem to the S3 filesystem. See Moving and Copying for more information.

ExtraArgs

S3 objects have additional properties, beyond a traditional filesystem. These options can be set using the upload_args and download_args properties. which are handed to upload and download methods, as appropriate, for the lifetime of the filesystem instance.

For example, to set the cache-control header of all objects uploaded to a bucket:

import fs, fs.mirror
miniofs = MINIOFS('example', upload_args={"CacheControl": "max-age=2592000", "ACL": "public-read"})
fs.mirror.mirror('/path/to/mirror', miniofs)

see the Boto3 docs for more information.

acl and cache_control are exposed explicitly for convenience, and can be used in URLs. It is important to URL-Escape the cache_control value in a URL, as it may contain special characters.

import fs, fs.mirror
with open fs.open_fs('s3://example?acl=public-read&cache_control=max-age%3D2592000%2Cpublic') as miniofs
    fs.mirror.mirror('/path/to/mirror', miniofs)

S3 URLs

You can get a public URL to a file on a S3 bucket as follows:

movie_url = miniofs.geturl('example.mov')

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for fs-miniofs, version 1.1.2
Filename, size File type Python version Upload date Hashes
Filename, size fs_miniofs-1.1.2-py2.py3-none-any.whl (10.6 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size fs-miniofs-1.1.2.tar.gz (10.4 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page