Tools to Detect Anomalous Events and News Related to Entities
Project description
huginn
Named for one of the Norse god Odin's ravens who would gather information for him. A package to facilitate the investigation of anomalous spikes in public interest in an entity, then fetch and summarize news stories relevant to those spikes.
Installation and Setup
Requires Python=3.6
huginn
is available on PyPi and installable via pip:
pip install huginn
Spacy models
Download English spacy models by running the following in the command line:
$ python3 -m spacy download en
NYT API Key
This project requires the usage of the New York Times article search API. Request a key at developer.nytimes.com. You'll need to set your NYT API key as a system environment variable. Open the file ~/.bashrc
(or ~/.bash_profile
if the .bashrc
doesn't exist) in any Unix based system (Mac, Ubuntu, etc), and add the following two lines to the end of the file:
# Setting NYT API Key as an environment variable
export NYT_API_KEY="YOUR_KEY_HERE"
Where YOUR_KEY_HERE
is your NYT API key, enclosed in quotes.
This is necessary for requesting relevant links for a potential entity's anomalies.
Usage
huginn is based around the Huginn
object. A user only need provide the entity that they are interested in.
from huginn import Huginn
point72 = Huginn('Point72')
At this point huginn fetches Google Trends search interest on this entity looking back to 2003. We can then find the most anomalous dates and plot the Google trends search interest with the anomalies:
point72.get_anomalies()
point72.plot_interest_with_anomalies()
We then fetch news articles and relevant information related to our anomalies:
point72.get_articles_info()
Now we can get summaries, either specific to each anomaly or globally:
point72.get_global_summary()
point72.get_articles_info_and_summary_after_LDA()
These require downloading some rather large summarization models to run.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.