s3kv 0.1.3

A Python library for interacting with S3 key-value store

S3KV - Simple Key-Value Store using AWS S3 and Elasticsearch

Introduction

S3KV is a Python library that provides a simple key-value store backed by S3 Object Storage and Elasticsearch for metadata indexing. It allows storing and retrieving JSON-formatted data with optional metadata for easy search and retrieval.

Initialization

To create an S3KV object, initialize it with the required parameters:

• bucket_name: The name of the S3 bucket to use for storing the data.
• aws_access_key_id and aws_secret_access_key: Optional AWS credentials for S3 access.
• elasticsearch_host: Optional Elasticsearch host URL if metadata indexing is needed.

Basic Operations

Add Data

• add(key, value, metadata=None): Adds a new key-value pair to the S3KV database. Optionally, metadata can be provided for indexing in Elasticsearch.

Retrieve Data

• get(key, default=None): Retrieves the value associated with the given key from the S3KV database. Optionally, a default value can be provided if the key does not exist.

Delete Data

• delete(key): Deletes a key-value pair from the S3KV database.

List Keys

• list_keys(): Lists all the keys in the S3KV database.

Check Key Existence

• key_exists(key): Checks if a key exists in the S3KV database.

Caching

S3KV supports caching of key-value pairs locally in the /tmp/s3kv_cache directory. Cached data can be retrieved using get_from_cache(key) and cleared using clear_cache() or clear_old_cache(max_days), which removes entries older than the specified days.

Tagging and Metadata

S3KV supports tagging keys with metadata, which can be useful for search and filtering purposes. Keys can be tagged individually using tag_key(key, tags) or in batches using tag_keys_with_prefix(prefix, tags).

Bucket Policy

To grant read and write access to the S3KV library, you can set a bucket policy using set_bucket_policy().

Example Usage

# Initialize S3KV with S3 bucket and optional Elasticsearch host
s3_kv = S3KV(bucket_name='my-s3-bucket', elasticsearch_host='http://localhost:9200')

# Add data
data = {'name': 'John', 'age': 30}
s3_kv.add(key='user1', value=data, metadata={'category': 'user'})

# Retrieve data
user_data = s3_kv.get(key='user1')

# Check key existence
if s3_kv.key_exists('user1'):
    print("User1 exists!")

# Delete data
s3_kv.delete('user1')

# List all keys
keys = s3_kv.list_keys()
print(keys)

Dependencies

• boto3: Python library for AWS S3 access
• elasticsearch: Python library for Elasticsearch access

Note

Ensure proper AWS credentials and Elasticsearch configurations are provided for proper functionality.