A tool to scrape data about fires from Twitter.
What is this?
This is a Python Twitter "Fire event" scraper/listener.
It is an application that will listen for or scrape data relating to house fires (Chicago specifically) in order to analyze how people use Twitter as a platform to report and talk about disasters.
How will this use Twitter data?
This application allows one to analyze, collect, and collate data about house fires and other disasters on Twitter.
How do I install this?
pip install twitter-fire-scraper
If it is already installed and a newer version is available, you can update with:
pip install twitter-fire-scraper --upgrade
This README assumes all commands take place in the same folder as this README file.
These should give you a good idea of how to use our scraper, and can be considered a 'living standard' of how our code works.
Setting up your secrets
This secrets file is only used for the demos. When using this library, it is up to you to manage how you store and retrieve your API keys.
More specifically, if the
Scraper object is not initialized with a
TwitterAuthentication object, it will search for a file called
~/.twitterfirescraper/secrets.json for API keys as a fallback.
This is to make the demonstrations work and not recommended usage when using the library.
A note: These are called 'secrets' for a reason. Don't ever stage or commit
You will need:
A twitter developer account & API key
- A consumer API key (goes into
- A consumer API secret key (goes into
- An access token (goes into
- An access secret (goes into
- A consumer API key (goes into
A twitter handle you're authorized to make queries on behalf of
You are to put these into a file called
secrets.json in your home folder under
.twitterfirescraper/ (For example, mine is
An example file is provided for you to base your file off of, called
The demos in our code connect to the following mongodb address:
Setting up a database
For the database, we have chosen to use MongoDB since twitter data is stored in JSON and MongoDB is very well-suited for storing JSON data.
Follow this tutorial on how to install MongoDB.
- Same as above.
- Ruby, used for running scripts to build and test Python wheels.
Setting up Pipenv
You can install Pipenv by executing
pip install pipenv
You can then install all packages (including dev packages like
twisted) in this folder's
./Pipenv by executing
pipenv install --dev
Then, you can run tests by executing
pipenv run python /src/twitter-fire-scraper/tests/test/__main__.py
Running a functional demo
Inside this folder, there are two files called
Run-Demo.ps1. You can run either of those to start a demo intended for presentation purposes.
Starting the Web API
There is a web API that is included with the
It exposes functions of the
twitter-fire-scraper over HTTP.
You can run the web API from the live source code with
pipenv run python twitter_fire_scraper/app.py.
You can run the Web API after installing it with
pip by typing
python -m twitter_fire_scraper.app
You can execute
pipenv run python fire-scraper/tests/<TESTNAME>.py to run a test.
To run all tests, execute
pipenv run python fire-scraper/tests/test/__init__.py and all tests will run.
Alternatively, if you have this package installed, run
python -m twitter_fire_scraper.tests.test
to run the package's test module.
What was this adapted from?
2fb844e8c081c1dc31cfb4760e3a80cefb6a0eee was used.
There's got to be a better way to run this than from the command line!
There is! Use an IDE (like PyCharm, which I use) that preferably integrates with Python to show you import errors, syntax errors, etc. Go google "Python IDE" and pick one you like.
Adding the location of Venv to your IDE
In order to run our tests through an IDE, we need to let our IDE know where venv was installed. I will explain this through Pycharm, but the method should be the same for any IDE.
python in windows powershell runs Python 3 (or you only have Python 3 installed), run
python -m pipenv --venv
This will yield the location of the python 3 Virtual Environment (It should be something like
C:\Users\Your Name\...\.virtualenvs\...). Copy this path and open Pycharm.
files -> settings and expand the
Project: fire-scraper-twitter. In the drop down, go into
Project Interpreter. Go to the top and click the gear and select
add, as we will be adding a new interpreter.
Existing environment and click the three dots to the right. Copy your path at the top, then OK everything.
There! Done! Now we can run our tests from inside our IDE.
Generating/uploading distribution archives
If you want to distribute this source code as a Python Wheel, follow this guide.
There are a series of Ruby scripts (cross-platform!) that handle building, cleaning, uploading.
Make sure you have the
twine package installed for Python.
You'll need to bump the version in
./VERSION when uploading a new version.
To the test site (test.pypi.org)
ruby upload.rb --test
To the real site (pypi.org)
ruby upload.rb --deploy
Testing download and install
There are a couple ways for you to test how a user would experience installing this package.
There are three Ruby scripts here, each doing what its name suggests.
test-localwheel-install.rb will install and test the latest WHL file generated by
test-testpypi-install.rb will install and test the TEST PyPI's
test-realpypi-install.rb will install and test the offical PyPI's
Release history Release notifications
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size twitter_fire_scraper-2.2.0-py3-none-any.whl (46.4 kB)||File type Wheel||Python version py3||Upload date||Hashes View hashes|
|Filename, size twitter_fire_scraper-2.2.0.tar.gz (34.8 kB)||File type Source||Python version None||Upload date||Hashes View hashes|
Hashes for twitter_fire_scraper-2.2.0-py3-none-any.whl
Hashes for twitter_fire_scraper-2.2.0.tar.gz