File-like interface for cloud block storage
Project description
Superfile
Simple Python file-like interface for cloud block storage. This is unlikely to be the most performant way do I/O with data from block storage, but it provides a uniform, simple API that can be used across multiple cloud block storage providers such as Google Cloud Storage, Amazon S3 and Azure Blob Storage.
Simple Usage
Importing the main module:
import superfile
Reading a text file:
with superfile.open('gs://my_bucket_name/file.txt', 'r') as f:
text = f.read()
Reading a binary file:
with superfile.open('gs://my_bucket_name/file.ext', 'rb') as f:
data = f.read()
Writing a text file:
text = 'hello world'
with superfile.open('gs://my_bucket_name/file.txt', 'w') as f:
f.write(text)
Writing a binary file:
data = b'hello world'
with superfile.open('gs://my_bucket_name/file.txt', 'wb') as f:
f.write(data)
Listing all files in a bucket:
fnames = list(superfile.list('gs://my_bucket_name'))
Listing files with a prefix from a bucket:
prefix = 'abc'
fnames = list(superfile.list(f'gs://my_bucket_name/{prefix}'))
Advanced Usage
Reading a file from GCS and providing auth credentials:
import superfile
from google.oauth2 import service_account
fpath = 'gs://my_bucket_name/file.txt'
creds = service_account.Credentials.from_service_account_file('/path/to/key.json')
with superfile.open(
path=fpath,
mode='r',
# All init_kwargs are passed to google.cloud.storage.Client.__init__().
init_kwargs=dict(credentials=creds),
# All open_kwargs are passed to google.cloud.storage.Blob.open().
open_kwargs=dict(errors='strict'),
) as f:
...
Reading a file from S3 and providing endpoint URL:
import superfile
fpath = 's3://my_bucket_name/file.txt'
endpoint_url = 'https://example.com'
with superfile.open(
path=fpath,
mode='r',
# All init_kwargs are passed to boto3.client().
init_kwargs=dict(endpoint_url=endpoint_url),
# All open_kwargs are passed to boto3.client.get_object() or boto3.client.put_object().
open_kwargs=dict(VersionId='...'),
) as f:
...
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for superfile-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e1231b44f6ed226399f33245674ba9614fda1c7a9a32ce52e150b4d3c868059 |
|
MD5 | 24cf00fc026b96f931c07086f7747a47 |
|
BLAKE2b-256 | 05c257b52df115e199c8493c6b2eecf4eaefc0479d1571a87dc0d7e3164e3090 |