A Python library for interacting with S3 key-value store
Project description
S3KV - Simple Key-Value Store using AWS S3 and Elasticsearch
Introduction
S3KV is a Python library that provides a simple key-value store backed by S3 Object Storage and Elasticsearch for metadata indexing. It allows storing and retrieving JSON-formatted data with optional metadata for easy search and retrieval.
Initialization
To create an S3KV object, initialize it with the required parameters:
bucket_name
: The name of the S3 bucket to use for storing the data.aws_access_key_id
andaws_secret_access_key
: Optional AWS credentials for S3 access.elasticsearch_host
: Optional Elasticsearch host URL if metadata indexing is needed.
Basic Operations
Add Data
add(key, value, metadata=None)
: Adds a new key-value pair to the S3KV database. Optionally, metadata can be provided for indexing in Elasticsearch.
Retrieve Data
get(key, default=None)
: Retrieves the value associated with the given key from the S3KV database. Optionally, a default value can be provided if the key does not exist.
Delete Data
delete(key)
: Deletes a key-value pair from the S3KV database.
List Keys
list_keys()
: Lists all the keys in the S3KV database.
Check Key Existence
key_exists(key)
: Checks if a key exists in the S3KV database.
Caching
S3KV supports caching of key-value pairs locally in the /tmp/s3kv_cache
directory. Cached data can be retrieved using get_from_cache(key)
and cleared using clear_cache()
or clear_old_cache(max_days)
, which removes entries older than the specified days.
Tagging and Metadata
S3KV supports tagging keys with metadata, which can be useful for search and filtering purposes. Keys can be tagged individually using tag_key(key, tags)
or in batches using tag_keys_with_prefix(prefix, tags)
.
Bucket Policy
To grant read and write access to the S3KV library, you can set a bucket policy using set_bucket_policy()
.
Example Usage
# Initialize S3KV with S3 bucket and optional Elasticsearch host
s3_kv = S3KV(bucket_name='my-s3-bucket', elasticsearch_host='http://localhost:9200')
# Add data
data = {'name': 'John', 'age': 30}
s3_kv.add(key='user1', value=data, metadata={'category': 'user'})
# Retrieve data
user_data = s3_kv.get(key='user1')
# Check key existence
if s3_kv.key_exists('user1'):
print("User1 exists!")
# Delete data
s3_kv.delete('user1')
# List all keys
keys = s3_kv.list_keys()
print(keys)
Dependencies
- boto3: Python library for AWS S3 access
- elasticsearch: Python library for Elasticsearch access
Note
Ensure proper AWS credentials and Elasticsearch configurations are provided for proper functionality.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.