Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Humilis kinesis stream processor plugin

Project Description

Kinesis event stream processor

A humilis plugin to deploy a Lambda function that maps events in a Kinesis event stream using a list of Python callables.

Installation

pip install humilis-kinesis-processor

Development

Assuming you have virtualenv installed:

make develop

Configure humilis:

.env/bin/humilis configure --local

Testing

Unit tests

To run the local test suite:

make test

Integration tests

Before running the integration test suite you need to set a few deployment secrets using the command:

s3keyring set [group]/[STAGE] [key] [secret]

In group humilis-kinesis-processor the following secrets need to be set:

By the default, the integration tests will deploy on a stage called DEV so the command to set the Sentry DSN is:

s3keyring set humilis-kinesis-processor:DEV sentry.dsn [SENTRYDSN]

To run the integration test suite:

make testi

The command above will deploy a Kinesis processor to your AWS account, and will also create additional resources (such as several Kinesis streams) needed to test that the deployment was successful. Once deployed, the integration tests will run, and once they have completed the test environment will be destroyed.

If you do not want the test environment to be destroyed after tests have completed you should run instead:

make testi DESTROY=no

You can also modify the name of the deployment stage by setting the STAGE environment variable. For instance, to deploy to a TEST stage:

make testi STAGE=TEST

More information

See humilis documentation.

Contact

If you have questions, bug reports, suggestions, etc. please create an issue on the GitHub project page.

License

This software is licensed under the MIT license

See License file

© 2016 Anatoly Bubenkov, German Gomez-Herrero, FindHotel and others.

Authors

Anatoly Bubenkov
initial idea and first alpha implementation
German Gomez-Herrero
actual implementation and further developments

Changelog

0.9.7

  • Support for Lambda environment variables

0.9.6

  • Do not silence exception if Kinesis payload is not a json object

0.9.5

  • Support for delivery streams filters and mappers

0.9.0

  • Better debugging of processing errors

0.8.9

  • Add error annotations to events that failed to be processed

0.8.6

  • Bugfix in error handling logic.

0.8.5

  • Better error handling.

0.7.7

  • Support for multiple delivery streams for input, error, and output(s)

0.7.6

  • Support for one-to-many mappers

0.7.5

  • Users can also specify input/output/error streams by name instead of as references to another layer outputs.

0.6.8

  • Set environment variables for the humilis environment/layer/stage

0.4.0

  • Major bugfix: permission to put records in delivery streams

0.3.0

  • Major bugfix: give permission to write state to DynamodDB

0.2.0

  • Package becomes a generic map-multiplex-map processor (germangh)

0.1.1

  • Cleanup package data (bubenkoff)

0.1.0

  • Major refactoring to simplify and clean the code (germangh)

0.0.1

  • Initial release (bubenkoff)
Release History

Release History

This version
History Node

1.3.6

History Node

1.3.5

History Node

1.3.4

History Node

1.3.3

History Node

1.3.2

History Node

1.3.1

History Node

1.3.0

History Node

1.2.9

History Node

1.2.8

History Node

1.2.7

History Node

1.2.6

History Node

1.2.5

History Node

1.2.4

History Node

1.2.3

History Node

1.2.2

History Node

1.2.1

History Node

1.1.9

History Node

1.1.8

History Node

1.1.7

History Node

1.1.6

History Node

1.1.5

History Node

1.1.4

History Node

1.1.3

History Node

1.1.2

History Node

1.1.1

History Node

1.1.0

History Node

1.0.9

History Node

1.0.8

History Node

1.0.7

History Node

1.0.6

History Node

1.0.5

History Node

1.0.4

History Node

1.0.3

History Node

1.0.2

History Node

1.0.1

History Node

1.0.0

History Node

0.9.9

History Node

0.9.8

History Node

0.9.7

History Node

0.9.5

History Node

0.9.4

History Node

0.9.3

History Node

0.9.2

History Node

0.9.1

History Node

0.9.0

History Node

0.8.9

History Node

0.8.8

History Node

0.8.7

History Node

0.8.4

History Node

0.8.3

History Node

0.8.2

History Node

0.8.1

History Node

0.8.0

History Node

0.7.9

History Node

0.7.8

History Node

0.7.7

History Node

0.7.6

History Node

0.7.4

History Node

0.7.3

History Node

0.7.2

History Node

0.7.1

History Node

0.7.0

History Node

0.6.9

History Node

0.6.8

History Node

0.6.7

History Node

0.6.6

History Node

0.6.5

History Node

0.6.4

History Node

0.6.3

History Node

0.6.2

History Node

0.6.1

History Node

0.6.0

History Node

0.5.0

History Node

0.4.0

History Node

0.3.0

History Node

0.2.2

History Node

0.2.0

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
humilis-kinesis-processor-1.3.6.tar.gz (14.1 kB) Copy SHA256 Checksum SHA256 Source Aug 11, 2017

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting