Skip to main content

distributed increment/decrement counter leveraging dynamodb

Project description

Build Status Documentation Status

Overview

distributed_counter is a python package for maintaining counters across clusters of computers. It leverages AWS DynamoDB for storage and atomicity of the counter.

Installation

distributed_counter is compatible with python2 and python3. Simply use pip to install.

pip install distributed_counter

You'll also want to set up your aws configurations.

Usage

The interface for distributed_counter is very simple. Everything is done through the DistributedCounter class. To instantiate:

from distributed_counter import DistributedCounter

counter = DistributedCounter('my_dynamo_table_name')

You can pass anything in kwargs to DistributedCounter and they will be propagated to the boto3 Session, e.g.

counter = DistributedCounter('mytable', region_name='us-west-1', aws_access_key_id='somekey',
                             aws_secret_access_key='somesecret')

There are also special parameters config and endpoint_url which get passed to the DynamoDB ServiceResource.

Table Creation

The dynamodb table has one HASH key called key. You can create the table yourself, or you can use create_table.

counter.create_table()

distributed_counter is smart enough to wait for the table to finish creation on your next call to the table.

Getting/Setting

Next, you can set a key using put and get it with get:

counter.put('mykey', 0)
counter.get('mykey')
0

Increment/Decrementing

Finally, you can increment/decrement.

counter.increment('mykey', 10)
10
counter.increment('mykey', 5)
15
counter.decrement('mykey', 14)
1

Note that the returned value is the new value.

You can use increment/decrement with a default. This is the same as doing a put then an increment/decrement.

counter.increment('nonexistantkey', 0, 0)
0
counter.increment('someotherkey', 0, 10)
10
counter.increment('yetanotherkey', 1, 10)
11

Example Use

Let's say we want to run a function every 100 calls to an API. In your API, you can put:

if not counter.increment('mykey', 1) % 100:
    run_function()
    counter.decrement('mykey', 100)

This guarantees that no matter how many servers you have, every 100 calls to your API will run your function.

The modulo is used instead of == because there is a corner case where hypothetically you could increment to 200 before the decrement finishes.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

distributed_counter-0.0.3.tar.gz (4.8 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page