Tweet Queue for AWS CDK
This is an AWS CDK construct library which allows you to get a feed of Twitter search results into an SQS queue. It works by periodically polling the freely available Twitter Standard Search API and sending all new tweets to an SQS queue.
Inspired by @jlhood's aws-serverless-twitter-event-source
Architecture
- A CloudWatch Event Rule triggers the poller AWS Lambda function periodically
- The poller reads the last checkpoint from a DynamoDB table (if exists)
- The poller issues a Twitter search query for all new tweets
- The poller enqueues all tweets to an SQS queue
- The poller stores the ID of the last tweet into the DynamoDB checkpoint table.
- Rinse & repeat.
Twitter API Keys
To issue a Twitter search request, you will need to apply for a Twitter developer account, and obtain API keys through by defining a new application.
The Twitter API keys are read by the poller from an AWS Secrets
Manager entry. The entry must contain
the following attributes: consumer_key, consumer_secret, access_token_key
and access_token_secret (exact names).
- Create a new AWS Secrets Manager entry for your API keys
- Fill in the key values as shown below:
- Store the key
- Obtain the ARN of the secret (you will need it soon).
Usage
Use npm to install the module in your CDK project. This will also add it to
your package.json file.
$ npm install cdk-tweet-queue
Add a TweetQueue to your CDK stack:
from cdk_tweet_queue import TweetQueue
queue = TweetQueue(self, "TweetStream",
# this is the ARN of the secret you stored
secret_arn="arn:aws:secretsmanager:us-east-1:1234567891234:secret:xxxxxxxxx",
# twitter search query
# see https://developer.twitter.com/en/docs/tweets/search/guides/standard-operators
query="#awscdk",
# optional properties
interval_min=60, # optional: polling interval in minutes
retention_period_sec=60, # optional: queue retention period
visibility_timeout_sec=60
)
Now, queue is an sqs.Queue object and can be used anywhere a queue is
accepted. For example, you could process the queue messages using an AWS Lambda
function by setting up an SQS event source mapping.
Development
The project is managed by projen and offers the following commands:
yarn projen- Synthesize the project configuration.yarn compile- Compile all source code.yarn test- Run all tests.yarn build- Complie, test, and package the module.
Integration test
There is also an integration test that can be executed by running the following commands. You will need to set the TWEET_QUEUE_SECRET_ARN environment variable in order for the test to be able to use your Twitter API keys.
$ yarn integ:deploy
Don't forget to destroy:
$ yarn integ:destroy
You can also run any cdk command on the integration test application by running:
yarn integ <command>
License
Apache-2.0
2 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file cdk_tweet_queue-2.0.962.tar.gz.
File metadata
- Download URL: cdk_tweet_queue-2.0.962.tar.gz
- Upload date:
- Size: 122.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.14.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
488d1df240dd500270fd540faf34d2640cd1f7763de8faf213a0978a7e9e13ca
|
|
| MD5 |
9fe490d48b265ba67277016d96795b20
|
|
| BLAKE2b-256 |
719c2f42dc8ae7fec7ae01e18d89a1b6decc9d362aa3219a000eaf013d3cc154
|
Provenance
The following attestation bundles were made for cdk_tweet_queue-2.0.962.tar.gz:
Publisher:
release.yml on cdklabs/cdk-tweet-queue
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
cdk_tweet_queue-2.0.962.tar.gz -
Subject digest:
488d1df240dd500270fd540faf34d2640cd1f7763de8faf213a0978a7e9e13ca - Sigstore transparency entry: 2336379856
- Sigstore integration time:
-
Permalink:
cdklabs/cdk-tweet-queue@c50cc36294a22246d3e5526661ace6c1a9a7cd09 -
Branch / Tag:
refs/heads/main - Owner: https://github.com/cdklabs
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@c50cc36294a22246d3e5526661ace6c1a9a7cd09 -
Trigger Event:
push
-
Statement type:
File details
Details for the file cdk_tweet_queue-2.0.962-py3-none-any.whl.
File metadata
- Download URL: cdk_tweet_queue-2.0.962-py3-none-any.whl
- Upload date:
- Size: 120.8 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/6.1.0 CPython/3.14.6
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ad6d84920ed0bc9f5d2e8b055db147a9bd3c564f98bc9f172f6f0cd08d780e68
|
|
| MD5 |
5b8317e3365560098bb0e859f436b318
|
|
| BLAKE2b-256 |
fbcf9bcc0483e7a0f1c077726ad8fa3e139f4b632f99241a9c675363367bcf05
|
Provenance
The following attestation bundles were made for cdk_tweet_queue-2.0.962-py3-none-any.whl:
Publisher:
release.yml on cdklabs/cdk-tweet-queue
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
cdk_tweet_queue-2.0.962-py3-none-any.whl -
Subject digest:
ad6d84920ed0bc9f5d2e8b055db147a9bd3c564f98bc9f172f6f0cd08d780e68 - Sigstore transparency entry: 2336379843
- Sigstore integration time:
-
Permalink:
cdklabs/cdk-tweet-queue@c50cc36294a22246d3e5526661ace6c1a9a7cd09 -
Branch / Tag:
refs/heads/main - Owner: https://github.com/cdklabs
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
release.yml@c50cc36294a22246d3e5526661ace6c1a9a7cd09 -
Trigger Event:
push
-
Statement type:
