pillaralgos 1.0.20

Algorithms for Pillar. Currently includes "mini" algorithms, nothing too sophisticated.

NOTE: This readme is just a quick reference. For more details include todo, near/medium/long term goals please see our GitHub page.

Table of Contents

1. Use
   1. Input variables
   2. Output variables
2. Background
   1. Algorithms
   2. Timeit Results
3. Build and Publis
4. Changelog

Use

To use any of the algorithms just import as needed with from pillaralgos import algo1, and then algo1(data, min_=2, save_json=False).

Input variables

save_json: bool
    True if want to save results as json to exports folder
data: list
min_: int
    Approximate number of minutes each clip should be
sort_by: str
    For algo1 ONLY
    'rel': "number of chatters at timestamp"/"number of chatters at that hour"
    'abs': "number of chatters at timestamp"/"total number of chatters in stream"
goal: str
    For algo3_5 ONLY
    'num_words': sum of the number of words in each chat message
    'num_emo': sum of the number of emoticons in each chat message
    'num_words_emo': sum of the number of words + emoticons in each chat message
min_words:int
    For algo3_0 ONLY
    When filtering chunks to top users, at least how many words the top user should send

Output variables

• All algorithms will return a result_json, list of dictionaries in the format of {start:seconds, end:seconds} where seconds is seconds elapsed since start of the stream. List is ordered from predicted best to worst timestamps.
• All algorithms can save the returned list as a .json if save_json=True is passed in.

Background

Pillar is creating an innovative way to automatically select and splice clips from Twitch videos for streamers. This repo is focusing on the algorithm aspect. Three main algorithms are being tested.

Algorithms

1. Algorithm 1: Find the best moments in clips based on where the most users participated. Most is defined as the ratio of unique users during a 2 min section to unique users for the entire session.
2. Algorithm 2 Find the best moments in clips based on when rate of messages per user peaked. This involves answering the question "at which 2 min segment do the most users send the most messages?". If users X, Y, and Z all send 60% of their messages at timestamp range delta, then that timestamp might qualify as a "best moment"
   1. NOTE: Currently answers the question "at which 2 min segment do users send the most messages fastest"
3. Algorithm 3 (WIP) Weigh each user by their chat rate, account age, etc. Heavier users predicted to chat more often at "best moment" timestamps
   1. STATUS: current weight determined by (num_words_of_user/num_words_of_top_user)
   2. Algorithm 3.5 Finds the best moments in clips based on most number of words/emojis/both used in chat

Timeit results

Results as of April 13, 2021 18:31 EST run on big_df with 80841 rows, 10 columns.

algo1	algo2	algo3_0	algo3_5
2.2 sec	1 min 23 sec	28.1 sec	16.3 sec

Build

To build and publish this package we are using the poetry python packager. It takes care of some background stuff that led to mistakes in the past.

Folder structure:

|-- dev_tools
    |-- pillaralgos_dev
        |-- __init__.py
        |-- dev_helpers.py # aws connection, file retrieval script
        |-- sanity_checks.py # placeholder
    |-- README.md 
    |-- pyproject.toml
|-- prod
    |-- pillaralgos  # <---- note that poetry didn't require an additional subfolder
        |-- helpers
            |-- __init__.py
            |-- data_handler.py
            |-- emoji_getter.py
        |-- __init__.py  # must include version number
        |-- algoXX.py  # all algorithms in separate files
        |-- brain.py
    |-- LICENSE
    |-- README.md
    |-- pyproject.toml  # must include version number
    |-- reinstall_pill.sh # quick script to uninstall local pillaralgos, build and install new one

To publish just run the poetry publish --build command after update version numbers as needed.

Changelog

• New algorithms
• Algo3.6: rank timestamps by emoji:user ratio
• Algo4: rank timestamps by compound score from SentimentAnalyzer
• Unit testing for algo 3.6