royston 0.0.14

A real time trend detection algorithm

Royston

An open source, real time trend detection framework written in Python. This project uses machine learning to detect trends in text over time.

Trends are identified by detecting phrases that start occurring much more frequently than those that don't typically occur. Various natural language processing and data science techniques are used to ensure similar words are modelled together (i.e. "cycle", "cycling" and "cyclist" all reduce down to a common word form, such as "cycle").

Documents can be grouped by a subject, so it is possible to detect "localised" trends. Similar phrases tend to relate to a particular trend, so hierachical clustering is used to make sure documents related to the same trend are grouped, rather than creating two "trends" about the same thing. For example, "doping scandal" and "Tour de France" are likely to be about the same thing...allegedly.

Based on ramekin, but going to take it further to do real time detection and maintaining models rather than creating them each time.

Installation and basic usage

We are going to create a royston to contain a set of news articles, and then find the trends.

First we will install the package via pip by typing the following into the command line:

pip3 install royston

Also install a couple of extra dependancies we will need for the example:

pip3 install datetime pytz

The following script creates some simple documents and adds them to a royston (also shipped in the examples directory):

from royston import Royston
from datetime import datetime as dt

roy = Royston()

# ingest a few documents
roy.ingest({ 'id': '123', 'body': 'Random text string', 'date': dt.now() })
roy.ingest({ 'id': '456', 'body': 'Antoher random string', 'date': dt.now() })

# find the trends - with this example, it won't find anything, as it's only got two stories!
trends = roy.trending()
print(trends)

Configuration Options

Constructor:

This package is heavily configurable to allow us to tune how we look for emerging trends. The default options have been set for the most common use case that looks at new trends that have emerged over the last 24 hours.

Currently, the main way of tuning these parameters is controlled by passing the Royston constructor an options dict with the following attributes:

Attribute	Type	Default	Description
min_trend_freq	int	4	A threshold for the minimum number of times a phrase has to occur in a single day before it can even be considered a trend for a given subject.
history_days	int	90	The context of the number of days to consider for the history. This means we look at how often a phrase has occured over this period, and get an idea of typical use.
trend_days	int	1	The period of time in which we want to look for trends. With the default of 1, we are looking at documents from the last day to see if new trends have emerged during that time compared with the typical use period defined by history_days
max_n	int	6	The maximum size of the n-gram window (i.e. the window size of each phrase)
keep_stops	Boolean	False	Keep stop words, based on the nltk stopword list
history_frequency_tolerance	float	1.6	Factor the history count by this amount to handle words that just didn't get mentioned in the history period. This usefulness of this is in review, and it is likely to be removed in future (or at least set to 1 by default).
trends_top_n	int	8	The maximum number of trends to return

Disclaimer: the following options are currently supported but expected to change significantly in future releases:

Attribute	Type	Default	Description
start	datetime	now - trend_days	The start of the "trend" period (i.e. a day ago)
end	datetime	now	The end of the "trend" period
history_start	datetime	start	Start of the trend period (i.e. history_days before end)
history_end	datetime	end - history_days	Start of the trend period (i.e. history_days before end)

Currently they are calculate in the constructor only, which is stupid, as we want this to run in realtime and adapt each time the trend method is called.

Running tests

poetry run test

Run coverage reports:

poetry run coverage

Distribute

This now uses poetry for package management, which can be done with the following command:

poetry build && poetry publish

Contribute?

This is still in the early stages of being ported over from JavaScript, and any help would be appreciated. The issues contain a lot of features that are needed. Please get in touch via LinkedIn and I can talk you thought anything.

Main concerns are:

• 100% test coverage.
• Retain the document format