abnosql 0.0.2

NoSQL Abstraction Library

NoSQL Abstraction Library

Basic CRUD and query support for NoSQL databases, allowing for portable cloud native applications

• AWS DynamoDB
• Azure Cosmos NoSQL

This library is not intended to create databases/tables, use Terraform/ARM/CloudFormation etc for that

Why not just use the name 'nosql' or 'pynosql'? because they already exist on pypi :-)

• NoSQL Abstraction Library
  • Installation
• Usage
  • API Docs
  • Querying
  • Indexes
  • Partition Keys
• Configuration
  • AWS DynamoDB
  • Azure Cosmos NoSQL
• Plugins and Hooks
• Testing
  • AWS DynamoDB
  • Azure Cosmos NoSQL
• CLI
• Future Enhancements / Ideas

Installation

pip install abnosql[dynamodb]
pip install abnosql[cosmos]

By default, abnosql does not include database depedendencies. This is to facilitate packaging abnosql into AWS Lambda or Azure Functions (for example), without over-bloating the packages

Usage

from abnosql import table
import os

os.environ['ABNOSQL_DB'] = 'dynamodb'

item = {
    'hk': '1',
    'rk': 'a',
    'num': 5,
    'obj': {
        'foo': 'bar',
        'num': 5,
        'list': [1, 2, 3],
    },
    'list': [1, 2, 3],
    'str': 'str'
}

tb = table('mytable')

tb.put_item(item)
tb.put_items([item])

# note partition/hash key should be first kwarg
assert tb.get_item(hk='1', rk='a') == item

assert tb.query({'hk': '1'})['items'] == [item]

# be careful not to use cloud specific statements!
assert tb.query_sql(
    'SELECT * FROM mytable WHERE hk = @hk AND num > @num',
    {'@hk': '1'}
)['items'] == [item]

tb.delete_item({'hk': '1', 'rk': 'a'})

API Docs

See API Docs

Querying

query() performs DynamoDB Query using KeyConditionExpression and exact match on FilterExpression if filters are supplied. For Cosmos, SQL is generated. This is the safest/most cloud agnostic way to query and probably OK for most use cases.

query_sql() performs Dynamodb ExecuteStatement passing in the supplied PartiQL statement. Cosmos uses the NoSQL SELECT syntax.

During mocked tests, SQLGlot is used to execute the statement, so results may differ...

Care should be taken with query_sql() to not to use SQL features that are specific to any specific provider (breaking the abstraction capability of using abnosql in the first place)

Indexes

Beyond partition and range keys defined on the table, indexes are not currently supported - and these will likey differ between providers anyway (eg DynamoDB supports Secondary Indexes, whereas Cosmos has Range, Spatial and Composite.

Partition Keys

A few methods such as get_item(), delete_item() and query() need to know partition/hash keys as defined on the table. To avoid having to configure this or lookup from the provider, the convention used is that the first kwarg or dictionary item is the partition key, and if supplied the 2nd is the range/sort key.

Configuration

AWS DynamoDB

Set the following environment variable and use the usual AWS environment variables that boto3 uses

• ABNOSQL_DB = "dynamodb"

Or set the boto3 session in the config

from abnosql import table
import boto3

tb = table(
    'mytable',
    config={'session': boto3.Session()},
    database='dynamodb'
)

Azure Cosmos NoSQL

Set the following environment variables

• ABNOSQL_DB = "cosmos"
• ABNOSQL_COSMOS_ACCOUNT = your database account
• ABNOSQL_COSMOS_ENDPOINT = drived from ABNOSQL_COSMOS_ACCOUNT if not set
• ABNOSQL_COSMOS_CREDENTIAL = your cosmos credential
• ABNOSQL_COSMOS_DATABASE = cosmos database

Or define in config

from abnosq import table

tb = table(
    'mytable',
    config={'account': 'foo', 'credential': 'someb64key', 'database': 'bar'},
    database='cosmos'
)

Plugins and Hooks

abnosql uses pluggy and registers in the abnosql.table namespace

The following hooks are available

• set_config - set config
• get_item_post - called after get_item(), can return modified data
• put_item_post
• put_items_post
• delete_item_post

See the TableSpecs and example test_hooks()

Testing

AWS DynamoDB

Use moto package and abnosql.mocks.mock_dynamodbx

mock_dynamodbx is used for query_sql and only needed if/until moto provides better partiql support

Example:

from abnosql.mocks import mock_dynamodbx 
from moto import mock_dynamodb

@mock_dynamodb
@mock_dynamodbx  # needed for query_sql only
def test_something():
    ...

More examples in tests/test_dynamodb.py

Azure Cosmos NoSQL

Use requests package and abnosql.mocks.mock_cosmos

Example:

from abnosql.mocks import mock_cosmos
import requests

@mock_cosmos
@responses.activate
def test_something():
    ...

More examples in tests/test_cosmos.py

CLI

Small abnosql CLI installed with few of the commands above

Usage: abnosql [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  delete-item
  get-item
  put-item
  put-items
  query
  query-sql

Example querying table in Azure Cosmos, with cosmos.json config file containing endpoint, credential and database

$ abnosql query-sql mytable 'SELECT * FROM mytable' -d cosmos -c cosmos.json
partkey      id      num  obj                                          list       str
-----------  ----  -----  -------------------------------------------  ---------  -----
p1           p1.1      5  {'foo': 'bar', 'num': 5, 'list': [1, 2, 3]}  [1, 2, 3]  str
p2           p2.1      5  {'foo': 'bar', 'num': 5, 'list': [1, 2, 3]}  [1, 2, 3]  str
p2           p2.2      5  {'foo': 'bar', 'num': 5, 'list': [1, 2, 3]}  [1, 2, 3]  str

Future Enhancements / Ideas

• test pagination & exception handling
• Google Firestore support, ideally in the core library (though could be added outside via use of the plugin system). Would need something like FireSQL implemented for oython, maybe via sqlglot
• Simple caching (maybe) using globals (used for AWS Lambda / Azure Functions)
• PostgresSQL support using JSONB column (see here for example). Would be nice to avoid an ORM and having to define a model for each table...
• blob storage backend? could use something similar to NoDB but maybe combined with smart_open and DuckDB's Hive Partitioning
• Redis..
• Hook implementations to write to ElasticSearch / OpenSearch for better searching. Useful when not able to use AWS Stream Processors Azure Change Feed, or Elasticstore. Why? because not all databases support stream processing, and if they do you don't want the hastle of using CDC
• database credential lookup using cloud native secret/vault services