Skip to main content

Easily load data from CSV to test out your DynamoDB table design

Project description

dynamodb-dev-importer (ddbimp)

Easily load data from CSV to test out your DynamoDB table design.

When working with DynamoDB, it is common practice to minimise the number of tables used, ideally down to just one.

Techniques such as sparse indexes and GSI overloading allow a lot of flexibility and efficiency.

Designing a good schema that supports your query patterns can be challenging. Often it is nice to try things out with a small amount of data. I personally find it convenient to enter data into a spreadsheet and play around with it there.

This utility eases the process of populating a DynamoDB table from a CSV file, exported from a spreadsheet, that follows a specific format common to DynamoDB modelling patterns.

Install

You can install it with

$ pip3 install ddbimp

Run

Assuming table people(pk:S, sk:S) is provisioned in your default region.

$ ddbimp --table people --skip 1 example.csv

Expected input format

pk sk data
PERSON-1 sales-Q1-2019 Alex jan: 12012 feb: 1927

Your spreadsheet (and exported CSV) should contain columns for:

  • pk
  • sk
  • data (optional)
  • anything after those three can contain arbitrary attributes of form attribute_name: value i.e. city: Edinburgh

Example row:

PERSON-1,sales-Q1-2019,Alex,jan: 12012,feb: 1927

Will yield an item like this:

{
    pk: 'PERSON-1',
    sk: 'sales-Q1-2019',
    data: 'Alex',
    jan: 12012,
    feb 1927
}

For a full example CSV, take a look at example.csv.

Key ideas

  • Table consists of partition key pk: S and sort key sk: S - their meaning varies depending on the item
  • A secondary index swaps the sort and partition keys, so the partition key is sk: S and sort key pk: S
  • A final secondary index uses sk: S and data: S where data is an arbitrary value you might want to search for, the meaning of data depends on the item it is part of
  • Group items through a shared partition key, store sub items with a sort key e.g.
    • e.g. pk:PERSON-1, sk:sales-Q1-2019, jan:12012, feb:1927

AWS recently released a preview build of a tool called NoSQL Workbench. It builds on the above ideas. I've tried it out and it seems pretty good, but I am a luddite and am faster working in a spreadsheet right now. I'd certainly recommend giving it a try.

Useful resources

Caveats, TODO

  • Uses your default AWS profile
  • Region needs to be set
  • Make work directly with a Google Sheets via sheets API

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for ddbimp, version 0.4
Filename, size File type Python version Upload date Hashes
Filename, size ddbimp-0.4-py3-none-any.whl (8.7 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size ddbimp-0.4.tar.gz (4.2 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page