Skip to main content
Python Software Foundation 20th Year Anniversary Fundraiser  Donate today!

A wrapper for select object operations using boto3 for Ceph which confirms integrity of reads and heals intermittent connectivity.

Project description

codecov

A wrapper for the following boto3 s3 client operations with connection retry and checksum verification:

  • get_object
  • head_object
  • upload_fileobj
  • copy_object
  • delete_object
  • list_objects_v2

Features

  • Retry connection failures
  • Confirm checksum of uploaded and retrieved objects
  • Move object
  • Constrained interface to support simple CRUD operations for objects in existing buckets

Configuration

The following environment variables configure the object clerk.

Variable Description Type Default
MULTIPART_THRESHOLD Threshold in bytes at which uploads are broken into multiple parts for upload. Impacts the checksum stored in the eTag STR 524288000
S3_CLIENT_CONFIG Boto Core Client Configuration https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html Defaults reflect capturing the retry of failed operations at a higher level. JSON `json {"connect_timeout": 60, "read_timeout": 60, "retries": {"max_attempts": 0}} `
S3_UPLOAD_CONFIG Transfer Configuration applied to uploads https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig multipart_threshold and multipart_chunksize are set by the MULTIPART_THRESHOLD environment variable. Values for these keys in the S3_UPLOAD_CONFIG will be discarded. JSON `json {} `
S3_DOWNLOAD_CONFIG Transfer Configuration applied to downloads https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig JSON `json {} `

Installation

pip install object-clerk

Examples

Initialize

clerk = ObjectClerk(host=127.0.0.1, port=8080, access_key=12342, secret_key=12342, retry_delay=1, retry_backoff=1, retry_jitter=(1, 3), retry_max_delay=5, retry_tries=3, use_ssl=False)'

Get Object

# with checksum verified

bytes_response = clerk.get_object("bucket", "object_key")

# without checksum verified

bytes_response = clerk.get_object("bucket", "object_key", verify_checksum=False)

Delete Object

clerk.delete_object("bucket", "object_key")

Get Object Info

dict_response = clerk.get_object_info("bucket", "object_key")

Copy Object

# with checksum verified

clerk.copy_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
)

# without checksum verified

clerk.copy_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
    verify_checksum=False
)

Upload Object

# with checksum verified

with open("file", mode='rb') as f:

    clerk.upload_object(f, "bucket", "object_key")

# without checksum verified

with open("file", mode='rb') as f:

    clerk.upload_object(f, "bucket", "object_key", verify_checksum=False)

Move Object

# with checksum verified

clerk.move_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
)

# without checksum verified

clerk.move_object(
    "source_bucket",
    "source_object_key",
    "destination_bucket",
    "destination_object_key",
    verify_checksum=False
)

List Object

clerk.list_objects(
    "bucket_name",
    1000
)

Test

git clone git@bitbucket.org:swiant/object_store_wrapper.git

pip install -e .

export HOST=<host>

export PORT=<port>

export ACCESS_KEY=<access_key>

export SECRET_KEY=<secret_key>

pytest -v object_clerk

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for object-clerk, version 0.0.12
Filename, size File type Python version Upload date Hashes
Filename, size object-clerk-0.0.12.tar.gz (15.4 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page