object-clerk 1.1.0

A wrapper for select object operations using boto3 for Ceph which confirms integrity of reads and heals intermittent connectivity.

A wrapper for the following boto3 s3 client operations with connection retry and checksum verification:

• get_object

• head_object

• upload_fileobj

• copy_object

• delete_object

• list_objects_v2

Features

• Retry connection failures

• Confirm checksum of uploaded and retrieved objects

• Move object

• Constrained interface to support simple CRUD operations for objects in existing buckets

Configuration

The following environment variables configure the object clerk.

Variable	Description	Type	Default
MULTIPART_THRESHOLD	Threshold in bytes at which uploads are broken into multiple parts for upload. Impacts the checksum stored in the eTag	STR	524288000
S3_CLIENT_CONFIG	Boto Core Client Configuration https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html Defaults reflect capturing the retry of failed operations at a higher level. May be overridden at initialization time with the client_config parameter.	JSON	`json {"connect_timeout": 60, "read_timeout": 60, "retries": {"max_attempts": 0}} `
S3_UPLOAD_CONFIG	Transfer Configuration applied to uploads https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig multipart_threshold and multipart_chunksize are set by the MULTIPART_THRESHOLD environment variable. Values for these keys in the S3_UPLOAD_CONFIG will be discarded. May be overridden at initialization time with the transfer_config parameter.	JSON	`json {} `
S3_DOWNLOAD_CONFIG	Reserved for download transfer configuration (currently not applied by object_clerk) https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig	JSON	`json {} `
CHECKSUM_RETRY_CONFIG	Configuration for retrying failed connections	JSON	`json {"retry_delay": 1, "retry_backoff": 1, "retry_jitter": [1, 3], "retry_max_delay": 5, "retry_tries": 3} `

Installation

pip install object-clerk

Examples

Initialize

clerk = ObjectClerk(host=127.0.0.1, port=8080, access_key=12342, secret_key=12342, retry_delay=1, retry_backoff=1, retry_jitter=(1, 3), retry_max_delay=5, retry_tries=3, use_ssl=False)'

Get Object

# with checksum verified

bytes_response = clerk.get_object("bucket", "object_key")

# without checksum verified

bytes_response = clerk.get_object("bucket", "object_key", verify_checksum=False)

Delete Object

clerk.delete_object("bucket", "object_key")

Get Object Info

dict_response = clerk.get_object_info("bucket", "object_key")

Copy Object

# with checksum verified

clerk.copy_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
)

# without checksum verified

clerk.copy_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
    verify_checksum=False
)

Upload Object

# with checksum verified

with open("file", mode='rb') as f:

    clerk.upload_object(f, "bucket", "object_key")

# without checksum verified

with open("file", mode='rb') as f:

    clerk.upload_object(f, "bucket", "object_key", verify_checksum=False)

Move Object

# with checksum verified

clerk.move_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
)

# without checksum verified

clerk.move_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
    verify_checksum=False
)

List Object

clerk.list_objects(
    "bucket_name",
    1000
)

Test

git clone [e-mail]:swiant/object_store_wrapper.git

pip install -e .

export HOST=<host>

export PORT=<port>

export ACCESS_KEY=<access_key>

export SECRET_KEY=<secret_key>

pytest -v object_clerk

To run the development tests locally, you need a running S3-compatible endpoint. The recommended approach is to use LocalStack via Docker:

# Start LocalStack S3 service
docker run --rm -d -p 4566:4566 -e SERVICES=s3 -e DEFAULT_REGION=us-east-1 localstack/localstack:latest

# Set environment variables for the test session
export OBJECT_STORE_HOST=localhost
export OBJECT_STORE_PORT=4566
export OBJECT_STORE_ACCESS_KEY=test
export OBJECT_STORE_SECRET_KEY=test
export AWS_REGION=us-east-1

# Run tests with tox (all environments)
tox
# Or run a specific environment
tox -e py313

# Stop LocalStack when done
docker stop $(docker ps -q --filter ancestor=localstack/localstack:latest)