Skip to main content

Common functionality for interacting with dynamodb easier. Most of the functionality is based on PynamoDB ORM library.

Project description

B.DynamoDbCommon

Pipeline

A python library that makes it easier to interact with AWS DynamoDB tables.

Description

This library extends functionality of pynamodb, boto3, json, etc. modules to make it easier to interact with AWS DynamoDB tables. It contains various useful functionalities:

  • Attributes (PynamoDB custom attributes);
  • Encoders (JSON encoders to work with pynamo/dynamo data types);
  • Models (PynamoDB custom models);
  • Seeds (Various data seeding functions);
  • Utils (Other cool functions);

Remarks

Biomapas aims to modernise life-science industry by sharing its IT knowledge with other companies and the community. This is an open source library intended to be used by anyone. Improvements and pull requests are welcome.

Related technology

  • Python3
  • Boto3
  • AWS DynamoDB
  • PynamoDB

Assumptions

This project assumes you know about DynamoDB service, and you prefer using PynamoDB ORM to interact with DynamoDB tables.

  • Good Python skills and basis of OOP.
  • Good PynamoDB/DynamoDB skills.

Useful sources

Install

Before installing this library, ensure you have these tools setup:

  • Python / Pip

To install this project from source run:

pip install .

Or you can install it from a PyPi repository:

pip install b-dynamodb-common

Usage & Examples

This section shows various examples on how to use this library.

Attributes

Attributes module. Contains various custom PynamoDB ORM attribtues.

  • Fernet attribute

Custom PynamoDB attribute that encrypts data in the database. Using Fernet algorithm

class User(Model):
    SECRET_ENCRYPTION_KEY = b'123456'

    first_name = FernetAttribute(SECRET_ENCRYPTION_KEY)
    last_name = FernetAttribute(SECRET_ENCRYPTION_KEY)
  • KMS attribute

Custom PynamoDB attribute that encrypts data in the database. Using AWS KMS key to encrypt/decrypt data.

boto_client = boto3.client('kms')
kms_key_arn = 'arn:of:the:custom:kms:key'

class User(Model):
    first_name = KmsAttribute(boto_client, kms_key_arn)
    last_name = KmsAttribute(boto_client, kms_key_arn)

Encoders

Encoders module. Contains various encoding functionality.

  • DynamoDbEncoder

Custom JSON encoder to handle DynamoDB data types.

data = {
    'key1': 'RandomData',
    'key2': OrderedSet([1, 2, 3]),
    'key3': Decimal(1.1)
}

json.dumps(data, cls=DynamoDbEncoder)
  • PynamoDbEncoder

Custom JSON encoder to handle PynamoDB ORM and DynamoDB data types.

data = {
    'key1': MapAttribute(map_key_1='RandomData'),
    'key2': OrderedSet([1, 2, 3]),
    'key3': Decimal(1.1)
}

json.dumps(data, cls=PynamoDbEncoder)

Models

Models module. Contains various PynamoDB-based custom models.

  • Permission model

Model that contains permissions attribute.

entity = PermissionsModel()
entity.pk = 'PK'
entity.permissions = ['list', 'of', 'permissions']
entity.save()

# Add more permissions and save.
entity.add_permission('permission')
entity.save()
  • Model type factory

Allows to use the same pynamodb Model against multiple databases.

from pynamodb.models import Model

# Create your own model. Example, User model.
class User(Model): pass

# Specify table 1 against which an example user will be saved.
user_model_table_1 = ModelTypeFactory(User).create('user_table_1', 'eu-west-1')
user_model_table_1(hash_key='hash', range_key='range').save()

# Specify another table and save user in different table.
user_model_table_2 = ModelTypeFactory(User).create('user_table_2', 'eu-east-1')
user_model_table_2(hash_key='hash', range_key='range').save()

Seeds

Currently this module is empty.

Utils

Utilities module. Contains lots of cool functions.

  • List function

Wraps PynamoDB query and scan functions for better management.

list_function: PynamoDBListFunction[DummyEntity] = PynamoDBListFunction(DummyEntity.query, 'PK')
items = list(list_function())

# You can also specify a transformer function to transform results before returning.
list_function: PynamoDBListFunction[DummyEntity] = PynamoDBListFunction(
    # PynamoDB list function (query).
    DummyEntity.query, 
    # Positional arguments.
    'PK',
    # Transformer function.
    transformer=lambda item: item.pk
)

# Will contain only pks.
ids = list(list_function())
  • List results

Wraps PynamoDB query and scan functions to handle recursive last_evaluated_key tokens.

list_function: PynamoDBListFunction[DummyEntity] = PynamoDBListFunction(
    DummyEntity.scan,
    limit=10,
    filter_condition=DummyEntity.pk.is_in([...])
)

result = PynamoDBListResult(list_function)

# Fetch one time.
result.fetch(recursive=False)

# Check whether all results have been fetched.
result.finished

# If not, feel free to call it one more time and not worry about last_evaluated_key.
result.fetch(recursive=False)

# If you want to retrieve absolutely all results in one call:
result.fetch(recursive=True) # Simple!

Testing

This package has integration tests based on pytest. To run tests simply run:

pytest b_dynamodb_common_test/integration/tests

Contribution

Found a bug? Want to add or suggest a new feature? Contributions of any kind are gladly welcome. You may contact us directly, create a pull-request or an issue in github platform. Lets modernize the world together.

Release history

0.4.0

  • Model type factory for dynamic table use now supports Indexes (global or local). Previously if you had e.g. GSI in your model, you would get a "Missing Meta" error.

0.3.0

  • Expose validate_permissions method.

0.2.0

  • Add model type factory to allow dynamic table_name and region specification.

0.1.0

  • Add transformer functions to transform returned results.
  • P.S. still not a stable version.

0.0.6

  • Add documentation.

0.0.5

  • Add Fernet and KMS attribute tests.

0.0.4

  • Add Dynamo and Pynamo encoders tests.

0.0.3

  • Add tests to test PermissionModel base class.

0.0.2

  • Improve pynamodb_list_result class.

0.0.1

  • Initial build.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

b_dynamodb_common-0.4.0.tar.gz (20.1 kB view details)

Uploaded Source

Built Distribution

b_dynamodb_common-0.4.0-py3-none-any.whl (25.1 kB view details)

Uploaded Python 3

File details

Details for the file b_dynamodb_common-0.4.0.tar.gz.

File metadata

  • Download URL: b_dynamodb_common-0.4.0.tar.gz
  • Upload date:
  • Size: 20.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.8.12

File hashes

Hashes for b_dynamodb_common-0.4.0.tar.gz
Algorithm Hash digest
SHA256 bb69f82dde09ac977c4fcf9fa4a5d530e120f0fdf1031bdedc9596b79dc568fc
MD5 c2d4fcc07989d84323110c9eabe93d60
BLAKE2b-256 7370d1beb35c087405aebd77bdf272b95b2cc4ec55b0104d3ddb9ba13eb499f4

See more details on using hashes here.

File details

Details for the file b_dynamodb_common-0.4.0-py3-none-any.whl.

File metadata

File hashes

Hashes for b_dynamodb_common-0.4.0-py3-none-any.whl
Algorithm Hash digest
SHA256 007ca794a8b27ce98b4c65f561895728d7591268559e0585c1a0ddee6a636f02
MD5 1ff7dcafcd1ae4a37f31b1e716ba8423
BLAKE2b-256 bb4f0a02917901a4a4e7a45ee7d6e030c8e41fbe9436b73e5c1503a37a6e51a9

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page