Skip to main content

A wrapper for select object operations using boto3 for Ceph which confirms integrity of reads and heals intermittent connectivity.

Project description

codecov

A wrapper for the following boto3 s3 client operations with connection retry and checksum verification:

  • get_object

  • head_object

  • upload_fileobj

  • copy_object

  • delete_object

  • list_objects_v2

Features

  • Retry connection failures

  • Confirm checksum of uploaded and retrieved objects

  • Move object

  • Constrained interface to support simple CRUD operations for objects in existing buckets

Configuration

The following environment variables configure the object clerk.

Variable

Description

Type

Default

MULTIPART_THRESHOLD

Threshold in bytes at which uploads are broken into multiple parts for upload. Impacts the checksum stored in the eTag

STR

524288000

S3_CLIENT_CONFIG

Boto Core Client Configuration https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html Defaults reflect capturing the retry of failed operations at a higher level.

JSON

`json {"connect_timeout": 60, "read_timeout": 60, "retries": {"max_attempts": 0}} `

S3_UPLOAD_CONFIG

Transfer Configuration applied to uploads https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig multipart_threshold and multipart_chunksize are set by the MULTIPART_THRESHOLD environment variable. Values for these keys in the S3_UPLOAD_CONFIG will be discarded.

JSON

`json {} `

S3_DOWNLOAD_CONFIG

Transfer Configuration applied to downloads https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig

JSON

`json {} `

Installation

pip install object-clerk

Examples

Initialize

clerk = ObjectClerk(host=127.0.0.1, port=8080, access_key=12342, secret_key=12342, retry_delay=1, retry_backoff=1, retry_jitter=(1, 3), retry_max_delay=5, retry_tries=3, use_ssl=False)'

Get Object

# with checksum verified

bytes_response = clerk.get_object("bucket", "object_key")

# without checksum verified

bytes_response = clerk.get_object("bucket", "object_key", verify_checksum=False)

Delete Object

clerk.delete_object("bucket", "object_key")

Get Object Info

dict_response = clerk.get_object_info("bucket", "object_key")

Copy Object

# with checksum verified

clerk.copy_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
)

# without checksum verified

clerk.copy_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
    verify_checksum=False
)

Upload Object

# with checksum verified

with open("file", mode='rb') as f:

    clerk.upload_object(f, "bucket", "object_key")

# without checksum verified

with open("file", mode='rb') as f:

    clerk.upload_object(f, "bucket", "object_key", verify_checksum=False)

Move Object

# with checksum verified

clerk.move_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
)

# without checksum verified

clerk.move_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
    verify_checksum=False
)

List Object

clerk.list_objects(
    "bucket_name",
    1000
)

Test

git clone git@bitbucket.org:swiant/object_store_wrapper.git

pip install -e .

export HOST=<host>

export PORT=<port>

export ACCESS_KEY=<access_key>

export SECRET_KEY=<secret_key>

pytest -v object_clerk

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

object-clerk-0.0.12.tar.gz (15.4 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page