Amazon simple storage service driver

This drb-driver-s3 module implements S3 protocol access with DRB data model.

S3 Factory and S3 Node

The module implements the factory model defined in DRB in its node resolver. Based on the python entry point mechanism, this module can be dynamically imported into applications.

The entry point group reference is drb.drivers.aws3.
The driver name is aws3.
The factory class is encoded into drb.drivers.aws3.
The S3 signature id is 4ab73f92-bbff-11ec-8422-0242ac120002.

Using this module

The project is present in https://www.pypi.org service. it can be freely loaded into projects with the following command line:

pip install drb-driver-s3

Access Data

DrbS3Node manages the s3 protocol to access remote data. The construction parameter is an authentication object.

from drb.drivers.aws3 import DrbS3Service, Auth
from botocore.config import Config

auth = Auth(service_name='s3',
            endpoint_url='http://your_s3_storage/',
            aws_access_key_id='user',
            aws_secret_access_key='password',
            config=Config(signature_version='s3v4'),
            region_name='us-east-1')
node = DrbS3Service(auth)

When accessing a DrbS3Service the node gives access to all the bucket of this service by giving a list of DrbS3Bucket, and then each node gives a list of DrbS3Object for each object in the bucket.

Resolving an s3:// URL

An s3://bucket/key URL resolves to:

• a DrbS3Service for s3:// (no bucket),
• a DrbS3Bucket for s3://bucket or any key ending with / (a prefix),
• a DrbS3Object for a key that is a real object.

When the key is not a real object, the factory raises DrbFactoryException (it probes existence with a HeadObject). This lets the DRB resolver walk the path segment by segment and descend an intermediate container instead of building a bogus object node whose first read would 404. It is what makes a key that continues inside an archive resolvable, e.g.:

from drb.topics.resolver import create
# descend into a zipped product on S3 and force the XML driver on a leaf
# that has no .xml extension:
node = create('s3+xml://bucket/product.zip/product.SAFE/manifest.safe')

The existence HeadObject is cached in the node context (node.context['resolver']['head']) so opening the object stream does not issue a second HeadObject.

GDAL locator (gdal-vsi)

A DrbS3Object exposes a GDAL /vsis3 locator through get_impl(str, 'gdal-vsi') (/vsis3/bucket/key) and, as a side effect, exposes the endpoint and credentials taken from the node's Auth to GDAL's /vsis3 handler via the standard AWS_* environment variables (no GDAL import in this driver). This lets a consumer such as drb-driver-image hand an object to GDAL for native range reads, and composes with the zip locator (/vsizip{/vsis3/bucket/key.zip}/inner.jp2). The GDAL S3 configuration is process-global, so a single credential set is assumed per process.

Limitations

This driver doesn't allow to write, modify, delete file on a s3 bucket, or it doesn't allow to delete or upload a file. This driver doesn't allow to download directly a bucket.

Documentation

The documentation of this implementation can be found here https://drb-python.gitlab.io/impl/aws3