Skip to main content

High performance, thread safe traversing tool for AWS DynamoDB

Project description

dynamodb-traverse

High performance, thread safe, hackable, general purpose traversing tool for AWS DynamoDB based on aioboto3.

Build Status Build Status

Why manually traverse dynamodb table?

There're tens of ways to consume dynamodb data, for example, dynamodb stream, emr dynamodb connector, kinesis stream... they are good for different use cases. Manual traverse has following benefits comparing to these solutions:

  • Deal with "small data"
  • Schema evolution, table migration
  • Custom TTL mechanism
  • Full control over offline traversing
  • Work with complicated nosql schema
  • Cross AWS account data replication/transformation

Irrelevant use cases

Since dynamodb-traverse is not native to AWS, do not use if your use cases like:

  • Real time streaming
  • Simple nosql schema that maps one primary key value to one sort key value
  • Big data (~TB) workload that requires dedicated emr clusters
  • Data backup

Installation/Uninstallation

Prerequisite: python 3.8+ and aioboto3>=6.4.1 (bleeding edge)

Run following command to install requirements:

$ pip install aioboto3

Next, install dynamodb-traverse by running:

$ pip install dynamodb-traverse

To uninstall dynamodb-traverse, run:

$ pip uninstall dynamodb-traverse

Setup

  • dynamodb-traverse by default looks at ~/.aws/credentials for profiles you specified in the client. Make sure you have created profile to access dynamodb.
  • You can specify audit log location when initializing client. By default it writes to /tmp/dynamodb_traverse_xxx.log.
  • We recommend using 35 as default scan batch size because of dynamodb limitations

Benchmark (in progress)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dynamodb-traverse-0.1.7.tar.gz (5.8 kB view details)

Uploaded Source

Built Distribution

dynamodb_traverse-0.1.7-py3-none-any.whl (10.6 kB view details)

Uploaded Python 3

File details

Details for the file dynamodb-traverse-0.1.7.tar.gz.

File metadata

  • Download URL: dynamodb-traverse-0.1.7.tar.gz
  • Upload date:
  • Size: 5.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0 requests-toolbelt/0.9.1 tqdm/4.38.0 CPython/3.8.0

File hashes

Hashes for dynamodb-traverse-0.1.7.tar.gz
Algorithm Hash digest
SHA256 d9a1ec7b751c8a25f5cdf34bb19f031952915ba8cd8fc71e3296bdf7206b8366
MD5 8e9f7d4d6f75e0e5507ab330398d7342
BLAKE2b-256 cb84daebea5445c8ccca0d555dff5da92a092e0605f5f3cd726bfaf47aae53ce

See more details on using hashes here.

File details

Details for the file dynamodb_traverse-0.1.7-py3-none-any.whl.

File metadata

  • Download URL: dynamodb_traverse-0.1.7-py3-none-any.whl
  • Upload date:
  • Size: 10.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0 requests-toolbelt/0.9.1 tqdm/4.38.0 CPython/3.8.0

File hashes

Hashes for dynamodb_traverse-0.1.7-py3-none-any.whl
Algorithm Hash digest
SHA256 e15d43b17f15db5c887284ed8ed7c2a401643676b3b58740e54f938452377f20
MD5 d0e0e2c454fa5376a89562bb53b6b7ea
BLAKE2b-256 c5e5034c38a9f53d3ceda6723e712c14823dcd84082a67b8f02c2c4e2094c9ce

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page