openS3-sdk 0.1.7

A boto3-like SDK for OpenS3

OpenS3 SDK

A boto3-compatible Python SDK for interacting with OpenS3, a local implementation of Amazon S3-like object storage functionality. This SDK provides a familiar interface for developers used to AWS boto3, making it easy to transition between AWS S3 and OpenS3.

⚠️ WARNING: You must have the OpenS3-server set up and running for this SDK to work properly. Please refer to the OpenS3-server repository for server setup instructions.

Table of Contents

• Features
• Installation
• Quick Start
• Client Configuration
• Bucket Operations
• Object Operations
• Error Handling
• Response Structure
• Examples
• Development
• Compatibility with boto3
• License

Features

• Boto3-compatible Interface: Familiar API for AWS developers
• Bucket Operations: Create, list, and delete buckets
• Object Operations: Upload, download, list, and delete objects
• File Utilities: Convenient methods for file uploads and downloads
• Flexible Authentication: Support for both AWS-style credentials and basic auth tuples
• Consistent Responses: Response structures match boto3 format

Installation

From PyPI (Recommended)

# Install from PyPI
pip install openS3-sdk

From Source

# Clone the repository
git clone https://github.com/SourceBox-LLC/OpenS3-SDK.git
cd opens3-sdk

# Create and activate a virtual environment (optional but recommended)
python -m venv venv
source venv/bin/activate  # On Windows: .\venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Install the SDK in development mode
pip install -e .

From GitHub

pip install git+https://github.com/SourceBox-LLC/OpenS3-SDK.git

Note: While the package name on PyPI is openS3-sdk, you'll still import it using import opens3 in your code.

Quick Start

import opens3

# Initialize a client
s3 = opens3.client('s3', 
                  endpoint_url='http://localhost:8000',
                  aws_access_key_id='admin',
                  aws_secret_access_key='password')

# Create a bucket
s3.create_bucket(Bucket='my-bucket')

# Upload a file
s3.upload_file('local_file.txt', 'my-bucket', 'remote_file.txt')

# Download a file
s3.download_file('my-bucket', 'remote_file.txt', 'downloaded_file.txt')

Client Configuration

Initialization

The OpenS3 client can be initialized with various configurations:

import opens3

# Basic initialization with defaults (uses admin/password auth)
s3 = opens3.client('s3', endpoint_url='http://localhost:8000')

# Using environment variables (recommended)
# Set OPENS3_ACCESS_KEY and OPENS3_SECRET_KEY environment variables
s3 = opens3.client('s3', endpoint_url='http://localhost:8000')

# Explicit credential parameters
s3 = opens3.client('s3',
                  endpoint_url='http://localhost:8000',
                  aws_access_key_id='admin',  # Also supports: access_key
                  aws_secret_access_key='password')  # Also supports: secret_key

# Direct auth tuple
s3 = opens3.client('s3',
                  endpoint_url='http://localhost:8000',
                  auth=('admin', 'password'))

Configuration Parameters

Parameter	Type	Description	Default
endpoint_url	str	URL to the OpenS3 server	'http://localhost:8000'
access_key	str	Username for authentication (from OPENS3_ACCESS_KEY env var)	'admin'
secret_key	str	Password for authentication (from OPENS3_SECRET_KEY env var)	'password'
aws_access_key_id	str	Alternate parameter name for username	None
aws_secret_access_key	str	Alternate parameter name for password	None
auth	tuple	Direct auth tuple (username, password)	None

Bucket Operations

Create a Bucket

response = s3.create_bucket(Bucket='my-bucket')

Parameters:

• Bucket: (Required) Name of the bucket to create

Response:

{
    'ResponseMetadata': {
        'HTTPStatusCode': 201
    },
    'Location': '/my-bucket'
}

List Buckets

response = s3.list_buckets()

Response:

{
    'Buckets': [
        {
            'Name': 'my-bucket',
            'CreationDate': datetime.datetime(2025, 5, 12, 17, 0, 0)
        },
        # More buckets...
    ],
    'Owner': {'ID': 'admin'}
}

Check if a Bucket Exists

response = s3.head_bucket(Bucket='my-bucket')

Parameters:

• Bucket: (Required) Name of the bucket to check

Returns:

• True if the bucket exists and the caller has permission to access it
• False if the bucket does not exist

Raises:

• HTTPError if the caller does not have permission to access the bucket (403) or other errors

Example:

try:
    if s3.head_bucket('my-bucket'):
        print("Bucket exists and you have access")
    else:
        print("Bucket does not exist")
except Exception as e:
    print(f"Error checking bucket: {e}")

Delete a Bucket

response = s3.delete_bucket(Bucket='my-bucket')

Parameters:

• Bucket: (Required) Name of the bucket to delete

Response:

{
    'ResponseMetadata': {
        'HTTPStatusCode': 200
    }
}

Object Operations

Upload an Object

Using put_object (Memory)

response = s3.put_object(
    Bucket='my-bucket',
    Key='hello.txt',
    Body=b'Hello World!'
)

Parameters:

• Bucket: (Required) Name of the bucket
• Key: (Required) Object key name
• Body: (Required) Object data - can be bytes or a file-like object

Response:

{
    'ResponseMetadata': {
        'HTTPStatusCode': 201
    },
    'ETag': '"fake-etag"'
}

Using upload_file (File)

response = s3.upload_file('local_file.txt', 'my-bucket', 'remote_file.txt')

Parameters:

• Filename: (Required) Path to the local file
• Bucket: (Required) Name of the bucket
• Key: (Required) Object key name in the bucket

List Objects

response = s3.list_objects_v2(
    Bucket='my-bucket',
    Prefix='folder/'  # Optional prefix
)

Parameters:

• Bucket: (Required) Name of the bucket
• Prefix: (Optional) Limit results to keys beginning with this prefix

Response:

{
    'Contents': [
        {
            'Key': 'folder/file1.txt',
            'LastModified': datetime.datetime(2025, 5, 12, 17, 0, 0),
            'Size': 12,
            'ETag': '"fake-etag"',
            'StorageClass': 'STANDARD'
        },
        # More objects...
    ],
    'Name': 'my-bucket',
    'Prefix': 'folder/',
    'MaxKeys': 1000,
    'KeyCount': 1,
    'IsTruncated': False
}

Download an Object

Using get_object (Memory)

response = s3.get_object(Bucket='my-bucket', Key='hello.txt')
content = response['Body'].content  # Access the binary content
text = content.decode('utf-8')      # Convert to text if needed

Parameters:

• Bucket: (Required) Name of the bucket
• Key: (Required) Object key name

Response:

{
    'Body': <Response object>,
    'ContentLength': 12,
    'LastModified': datetime.datetime(2025, 5, 12, 17, 0, 0),
    'ContentType': 'text/plain'
}

Using download_file (File)

response = s3.download_file('my-bucket', 'hello.txt', 'downloaded.txt')

Parameters:

• Bucket: (Required) Name of the bucket
• Key: (Required) Object key name
• Filename: (Required) Path where the file should be saved

Delete an Object

response = s3.delete_object(Bucket='my-bucket', Key='hello.txt')

Parameters:

• Bucket: (Required) Name of the bucket
• Key: (Required) Object key name

Response:

{
    'ResponseMetadata': {
        'HTTPStatusCode': 200
    }
}

Error Handling

The SDK provides boto3-compatible error handling with enhanced error messages. Starting from version 0.1.6, the SDK includes detailed error information from the server to help with debugging and user feedback.

Basic Error Handling

from opens3.exceptions import ClientError, NoSuchBucket, NoSuchKey

try:
    s3.get_object(Bucket='non-existent-bucket', Key='file.txt')
except NoSuchBucket as e:
    print(f"Bucket does not exist: {e}")
except NoSuchKey as e:
    print(f"Object does not exist: {e}")
except ClientError as e:
    print(f"Error: {e.response['Error']['Message']}")

Enhanced Error Details (v0.1.6+)

In version 0.1.6 and above, HTTP errors include additional details extracted from the server response:

import requests

try:
    # Try to delete a non-empty bucket
    s3.delete_bucket(Bucket='my-bucket-with-objects')
except requests.exceptions.HTTPError as e:
    # Access detailed error message from the server
    if hasattr(e, 'detail'):
        print(f"Detailed error: {e.detail}")
    # Access the original status code
    if hasattr(e, 'status_code'):
        print(f"Status code: {e.status_code}")
    # General error handling
    print(f"Error: {str(e)}")

This provides more specific information about why operations failed, such as detailed reasons for bucket deletion failures.

Response Structure

OpenS3 SDK responses mirror the structure of boto3 responses to provide compatibility. Here's a breakdown of common response elements:

• ResponseMetadata: Contains metadata about the request, including HTTPStatusCode
• Operation-specific data (e.g., Buckets for list_buckets, Contents for list_objects_v2)

Examples

Working with Buckets

import opens3

s3 = opens3.client('s3', endpoint_url='http://localhost:8000')

# Create multiple buckets
s3.create_bucket(Bucket='bucket1')
s3.create_bucket(Bucket='bucket2')

# List all buckets
response = s3.list_buckets()
for bucket in response['Buckets']:
    print(f"Bucket: {bucket['Name']}, Created: {bucket['CreationDate']}")

# Clean up - delete buckets
s3.delete_bucket(Bucket='bucket1')
s3.delete_bucket(Bucket='bucket2')

Working with Objects

import opens3

s3 = opens3.client('s3', endpoint_url='http://localhost:8000')

# Create a bucket for our objects
s3.create_bucket(Bucket='files')

# Upload multiple objects
s3.put_object(Bucket='files', Key='doc1.txt', Body=b'Document 1 content')
s3.put_object(Bucket='files', Key='doc2.txt', Body=b'Document 2 content')
s3.put_object(Bucket='files', Key='folder/doc3.txt', Body=b'Document 3 content')

# List all objects
response = s3.list_objects_v2(Bucket='files')
print(f"All objects in bucket 'files':")
for obj in response['Contents']:
    print(f"- {obj['Key']} ({obj['Size']} bytes)")

# List objects with prefix
response = s3.list_objects_v2(Bucket='files', Prefix='folder/')
print(f"\nObjects in 'folder/':")
for obj in response['Contents']:
    print(f"- {obj['Key']}")

# Download an object
response = s3.get_object(Bucket='files', Key='doc1.txt')
print(f"\nContent of doc1.txt: {response['Body'].content.decode('utf-8')}")

Development

Running Tests

# Install test dependencies
pip install pytest

# Run the test suite
python -m pytest

Building the Package

pip install build twine
python -m build

Compatibility with boto3

This SDK aims to provide a compatible interface with boto3 for the most common S3 operations. It's designed to make it easy for developers to switch between AWS S3 and OpenS3 with minimal code changes.

Some notable differences:

• Only core S3 operations are currently supported
• Some advanced features like multipart uploads and presigned URLs are not yet implemented
• The resource interface (opens3.resource()) is not yet available

License

MIT