Stream Tweets into Your Favorite Databases
Project description
twimer
Stream Tweets into Your Favorite Databases
Analyzing tweets reveals very interesting insights about events in a specific time and location, people's opinions about the news, etc. twimer aims to make the data collection easier for you so you can focus on your data analysis only! This tool stores tweets with certain keywords and from specific geographic regions into JSON files or MongoDB databases.
Twitter API
To use twimer, you need to obtain CONSUMER_KEY, CONSUMER_SECRET, ACCESS_TOKEN, ACCESS_TOKEN_SECRET from Twitter Developer. You can either set these as environment variables or directly pass them to twimer.
Installation
Simply install this package by running the following command:
pip install twimer
Usage
In the current version, you can stream the tweets using keywords and store them in files (JSON and JSON.tar.gz) and MongoDB databases.
- To store tweets that contain keyword1 and keyword2 keywords as JSON (tar.gz) files into the
my_path
directory:
import twimer
stream_tweet = twimer.Twimer(CONSUMER_KEY,
CONSUMER_SECRET,
ACCESS_TOKEN,
ACCESS_TOKEN_SECRET,
storage_method='targz',
file_path=my_path)
stream_tweet.start_streaming(filters=['keyword1', 'keyword2'])
- To store the tweets in a MongoDB database using url
mongo_url
:
import twimer
stream_tweet = twimer.Twimer(CONSUMER_KEY,
CONSUMER_SECRET,
ACCESS_TOKEN,
ACCESS_TOKEN_SECRET,
storage_method='mongodb',
mongo_url=mongo_url)
stream_tweet.start_streaming(filters=['keyword1', 'keyword2'])
The my_url
is in mongodb://[username:password@]host1[:port1][,...hostN[:portN]][/[defaultauthdb][?options]] format. If you set your Twitter credentials as environment variables, you do not need to pass them and can simply run the following code.
import twimer
stream_tweet = twimer.Twimer(storage_method='targz',
file_path=my_path)
stream_tweet.start_streaming(filters=['keyword1', 'keyword2'])
Local MongoDB Database
To store tweets using MongoDB locally, you can use docker-compose by running docker-compose up
. Note that you need to install docker on your system before running this command. To view the MongoDB Express dashboard, simply open http://localhost:8081
on your browser. The default username and password for MongoDB are set to root and example and you need to update them to prevent unauthorized access to your local database.
Contribution
You are very welcome to contribute to this project with your code (as pull-requests), mention the bugs, or ask for new features (as GitHub Issues), or just tell your friends about it!
You can also directly contact me by email.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.