object-clerk 0.1.1

A wrapper for select object operations using boto3 for Ceph which confirms integrity of reads and heals intermittent connectivity.

A wrapper for the following boto3 s3 client operations with connection retry and checksum verification:

• get_object

• head_object

• upload_fileobj

• copy_object

• delete_object

• list_objects_v2

Features

• Retry connection failures

• Confirm checksum of uploaded and retrieved objects

• Move object

• Constrained interface to support simple CRUD operations for objects in existing buckets

Configuration

The following environment variables configure the object clerk.

Variable	Description	Type	Default
MULTIPART_THRESHOLD	Threshold in bytes at which uploads are broken into multiple parts for upload. Impacts the checksum stored in the eTag	STR	524288000
S3_CLIENT_CONFIG	Boto Core Client Configuration https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html Defaults reflect capturing the retry of failed operations at a higher level.	JSON	`json {"connect_timeout": 60, "read_timeout": 60, "retries": {"max_attempts": 0}} `
S3_UPLOAD_CONFIG	Transfer Configuration applied to uploads https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig multipart_threshold and multipart_chunksize are set by the MULTIPART_THRESHOLD environment variable. Values for these keys in the S3_UPLOAD_CONFIG will be discarded.	JSON	`json {} `
S3_DOWNLOAD_CONFIG	Transfer Configuration applied to downloads https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig	JSON	`json {} `

Installation

pip install object-clerk

Examples

Initialize

clerk = ObjectClerk(host=127.0.0.1, port=8080, access_key=12342, secret_key=12342, retry_delay=1, retry_backoff=1, retry_jitter=(1, 3), retry_max_delay=5, retry_tries=3, use_ssl=False)'

Get Object

# with checksum verified

bytes_response = clerk.get_object("bucket", "object_key")

# without checksum verified

bytes_response = clerk.get_object("bucket", "object_key", verify_checksum=False)

Delete Object

clerk.delete_object("bucket", "object_key")

Get Object Info

dict_response = clerk.get_object_info("bucket", "object_key")

Copy Object

# with checksum verified

clerk.copy_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
)

# without checksum verified

clerk.copy_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
    verify_checksum=False
)

Upload Object

# with checksum verified

with open("file", mode='rb') as f:

    clerk.upload_object(f, "bucket", "object_key")

# without checksum verified

with open("file", mode='rb') as f:

    clerk.upload_object(f, "bucket", "object_key", verify_checksum=False)

Move Object

# with checksum verified

clerk.move_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
)

# without checksum verified

clerk.move_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
    verify_checksum=False
)

List Object

clerk.list_objects(
    "bucket_name",
    1000
)

Test

git clone [e-mail]:swiant/object_store_wrapper.git

pip install -e .

export HOST=<host>

export PORT=<port>

export ACCESS_KEY=<access_key>

export SECRET_KEY=<secret_key>

pytest -v object_clerk