Skip to main content

High performance, thread safe traversing tool for AWS DynamoDB

Project description

dynamodb-traverse

High performance, thread safe, hackable, general purpose traversing tool for AWS DynamoDB based on aioboto3.

Why manually traverse dynamodb table?

There're tens of ways to consume dynamodb data, for example, dynamodb stream, emr dynamodb connector, kinesis stream... they are good for different use cases. Manual traverse has following benefits comparing to these solutions:

  • Deal with "small data"
  • Schema evolution, table migration
  • Custom TTL mechanism
  • Full control over offline traversing
  • Work with complicated nosql schema
  • Cross AWS account data replication/transformation

Irrelevant use cases

Since dynamodb-traverse is not native to AWS, do not use if your use cases like:

  • Real time streaming
  • Simple nosql schema that maps one primary key value to one sort key value
  • Big data (~TB) workload that requires dedicated emr clusters
  • Data backup

Installation/Uninstallation

Prerequisite: python 3.8+ and aioboto3>=6.4.1 (bleeding edge)

Run following command to install requirements:

$ pip install aioboto3

Next, install dynamodb-traverse by running:

$ pip install dynamodb-traverse

To uninstall dynamodb-traverse, run:

$ pip uninstall dynamodb-traverse

Setup

  • dynamodb-traverse by default looks at ~/.aws/credentials for profiles you specified in the client. Make sure you have created profile to access dynamodb.
  • You can specify audit log location when initializing client. By default it writes to /tmp/dynamodb_traverse_xxx.log.
  • We recommend using 35 as default scan batch size because of dynamodb limitations

Benchmark (in progress)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for dynamodb-traverse, version 0.1.7
Filename, size File type Python version Upload date Hashes
Filename, size dynamodb_traverse-0.1.7-py3-none-any.whl (10.6 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size dynamodb-traverse-0.1.7.tar.gz (5.8 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page