File-like interface for cloud block storage
Project description
Superfile
Simple Python file-like interface for cloud block storage. This is unlikely to be the most performant way do I/O with data from block storage, but it provides a uniform, simple API that can be used across multiple cloud block storage providers such as Google Cloud Storage, Amazon S3 and Azure Blob Storage.
Simple Usage
Importing the main module:
import superfile
Reading a local file:
with superfile.open('my/local/path', 'r') as f:
contents = f.read()
Reading a text file:
with superfile.open('gs://my_bucket_name/file.txt', 'r') as f:
text = f.read()
Reading a binary file:
with superfile.open('gs://my_bucket_name/file.ext', 'rb') as f:
data = f.read()
Writing a text file:
text = 'hello world'
with superfile.open('gs://my_bucket_name/file.txt', 'w') as f:
f.write(text)
Writing a binary file:
data = b'hello world'
with superfile.open('gs://my_bucket_name/file.txt', 'wb') as f:
f.write(data)
Listing all files in a bucket:
fnames = list(superfile.list('gs://my_bucket_name'))
Listing files with a prefix from a bucket:
prefix = 'abc'
fnames = list(superfile.list(f'gs://my_bucket_name/{prefix}'))
Advanced Usage
Reading a file from GCS and providing auth credentials:
import superfile
from google.oauth2 import service_account
fpath = 'gs://my_bucket_name/file.txt'
creds = service_account.Credentials.from_service_account_file('/path/to/key.json')
with superfile.open(
path=fpath,
mode='r',
# All init_kwargs are passed to google.cloud.storage.Client.__init__().
init_kwargs=dict(credentials=creds),
# All open_kwargs are passed to google.cloud.storage.Blob.open().
open_kwargs=dict(errors='strict'),
) as f:
...
Reading a file from S3 and providing endpoint URL:
import superfile
fpath = 's3://my_bucket_name/file.txt'
endpoint_url = 'https://example.com'
with superfile.open(
path=fpath,
mode='r',
# All init_kwargs are passed to boto3.client().
init_kwargs=dict(endpoint_url=endpoint_url),
# All open_kwargs are passed to boto3.client.get_object() or boto3.client.put_object().
open_kwargs=dict(VersionId='...'),
) as f:
...
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for superfile-0.0.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e5bd32a37e1008dc5ed4cf94da45ee556e377457b7439422d82ab96b79721423 |
|
MD5 | 5af0b39b341da1443c8cd4d8ea59497b |
|
BLAKE2b-256 | dcc4a04ce7ee19e08a996fb1d630352800ab836707c7c1b331bf847c4f99fc03 |