Defines an SQS queue with tweet stream from a search
Tweet Queue for AWS CDK
This is an AWS CDK construct library which allows you to get a feed of Twitter search results into an SQS queue. It works by periodically polling the freely available Twitter Standard Search API and sending all new tweets to an SQS queue.
- A CloudWatch Event Rule triggers the poller AWS Lambda function periodically
- The poller reads the last checkpoint from a DynamoDB table (if exists)
- The poller issues a Twitter search query for all new tweets
- The poller enqueues all tweets to an SQS queue
- The poller stores the ID of the last tweet into the DynamoDB checkpoint table.
- Rinse & repeat.
Twitter API Keys
The Twitter API keys are read by the poller from an AWS Secrets
Manager entry. The entry must contain
the following attributes:
access_token_secret (exact names).
- Create a new AWS Secrets Manager entry for your API keys
- Fill in the key values as shown below:
- Store the key
- Obtain the ARN of the secret (you will need it soon).
npm to install the module in your CDK project. This will also add it to
$ npm install cdk-tweet-queue
TweetQueue to your CDK stack:
# Example automatically generated without compilation. See https://github.com/aws/jsii/issues/826 from cdk_tweet_queue import TweetQueue queue = TweetQueue(self, "TweetStream", # this is the ARN of the secret you stored secret_arn="arn:aws:secretsmanager:us-east-1:1234567891234:secret:xxxxxxxxx", # twitter search query # see https://developer.twitter.com/en/docs/tweets/search/guides/standard-operators query="#awscdk", # optional properties interval_min=60, # optional: polling interval in minutes retention_period_sec=60, # optional: queue retention period visibility_timeout_sec=60 )
queue is an
sqs.Queue object and can be used anywhere a queue is
accepted. For example, you could process the queue messages using an AWS Lambda
function by setting up an SQS event source mapping.
This is a mono-repo which uses lerna. Here are some useful commands:
lerna run build- builds all code
lerna run watch --stream-- runs
tsc -win all modules (in parallel)
lerna run test- tests all code
There is also an integration test that can be executed from the cdk-tweet-queue
package by running the following commands. You will need to set the
TWEET_QUEUE_SECRET_ARN environment variable in order for the test to be able
to use your Twitter API keys.
$ npm run integ deploy ...
Don't forget to destroy:
$ npm run integ destroy ...
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size cdk_tweet_queue-1.0.3-py3-none-any.whl (1.1 MB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size cdk-tweet-queue-1.0.3.tar.gz (1.1 MB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for cdk_tweet_queue-1.0.3-py3-none-any.whl