Skip to main content
Help us improve PyPI by participating in user testing. All experience levels needed!

No project description provided

Project description

This library offers some functionality to assist in writing records to AWS services in batches, where your data is not naturally batched. This helps to achieve significant efficiencies when interacting with those AWS services as batch writes are often times orders of magnitude faster than individual writes.

## General Usage The library is very simple to use and works on the concept that you wish to submit payloads to a dispatcher, and not care too much about how to batch up the payloads and send them into their target service. This allows you to utilise the significant efficiencies of patch send/put/write methods, without the problem of error handling and batch sizes.

Each of the supported services has it’s own dispatcher class. Each has the same 2 meths with which to interact: * submit_payload: pass in a payload (e.g. a single message, metric etc) * flush_payloads: send all payloads in the backlog. Don’t worry if there are a lot of payloads, the dispatcher will automatically break them down into chunks.

The `flush_payloads` method does nto need to be called. All records will be flushed when you program does gc or shuts down. However, it is recommended you flush records every invocation or when payload lists become long (100-200 items +) to avoid issues.

## Supported Services

#### Kinesis Put Batch Put items to a Kinesis stream `python from boto3_batch_utils import KinesisBatchDispatcher kn = KinesisBatchDispatcher('MyExampleStreamName') kn.submit_payload({"something": "in", "my": "message"}) kn.submit_payload({"tells": "me", "this": "is", "easy": True}) kn.flush_payloads() `

#### Dynamo Write Batch write records to a DynamoDB table `python from boto3_batch_utils import DynamoBatchDispatcher dy = DynamoBatchDispatcher('MyExampleDynamoTable', primary_partition_key='Id') dy.submit_payload({"something": "in", "my": "message"}) dy.submit_payload({"tells": "me", "this": "is", "easy": True}) dy.flush_payloads() `

#### Cloudwatch Put Metrics Batch put metric data to Cloudwatch. CLoudwatch comes with a handy dimension builder function cloudwatch_dimension to help you construct dimensions `python from boto3_batch_utils import CloudwatchBatchDispatcher, cloudwatch_dimension cw = CloudwatchBatchDispatcher('TEST_JUNK') cw.submit_payload('RANDOM_METRIC', dimensions=cloudwatch_dimension('dimA', '12345'), value=555, unit='Count') cw.submit_payload('RANDOM_METRIC', dimensions=cloudwatch_dimension('dimA', '12345'), value=1234, unit='Count') cw.flush_payloads() `

#### SQS Send Messages Batch send messages to an SQS queue `python from boto3_batch_utils import SQSBatchDispatcher sqs = SQSBatchDispatcher("aQueueWithAName") sqs.submit_payload("some message of some sort") sqs.submit_payload("a different message, probably a similar sort") sqs.flush_payloads() `

Project details


Release history Release notifications

History Node

1.4.0

History Node

1.3.0

History Node

1.2.4

History Node

1.2.3

History Node

1.2.2

History Node

1.2.1

History Node

1.2.0

History Node

1.1.6

History Node

1.1.5

History Node

1.1.4

History Node

1.1.3

History Node

1.1.2

History Node

1.1.1

History Node

1.1.0

History Node

1.0.29

History Node

1.0.28

History Node

1.0.27

History Node

1.0.26

History Node

1.0.25

History Node

1.0.24

History Node

1.0.23

History Node

1.0.22

History Node

1.0.21

History Node

1.0.20

History Node

1.0.19

History Node

1.0.18

History Node

1.0.17

History Node

1.0.16

History Node

1.0.15

History Node

1.0.14

History Node

1.0.13

History Node

1.0.12

History Node

1.0.11

This version
History Node

1.0.10

History Node

1.0.9

History Node

1.0.8

History Node

1.0.7

History Node

1.0.6

History Node

1.0.5

History Node

1.0.4

History Node

1.0.3

History Node

1.0.2

History Node

1.0.1

History Node

1.0.0

History Node

0.2.14

History Node

0.2.13

History Node

0.2.12

History Node

0.2.11

History Node

0.2.10

History Node

0.2.9

History Node

0.2.8

History Node

0.2.7

History Node

0.2.6

History Node

0.2.5

History Node

0.2.4

History Node

0.2.3

History Node

0.2.2

History Node

0.2.1

History Node

0.2.0

History Node

0.1.5

History Node

0.1.3

History Node

0.1.2

History Node

0.1.1

History Node

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
boto3_batch_utils-1.0.10.tar.gz (19.9 kB) Copy SHA256 hash SHA256 Source None Mar 13, 2018

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page