Algorithms for Pillar. Currently includes "mini" algorithms, nothing too sophisticated.
Project description
NOTE: This readme is just a quick reference. For more details include todo, near/medium/long term goals please see our GitHub page.
Table of Contents
Use
To use any of the algorithms just import as needed with from pillaralgos import algo1
, and then algo1(data, min_=2, save_json=False)
.
Input variables
save_json: bool
True if want to save results as json to exports folder
data: list
min_: int
Approximate number of minutes each clip should be
sort_by: str
For algo1 ONLY
'rel': "number of chatters at timestamp"/"number of chatters at that hour"
'abs': "number of chatters at timestamp"/"total number of chatters in stream"
goal: str
For algo3_5 ONLY
'num_words': sum of the number of words in each chat message
'num_emo': sum of the number of emoticons in each chat message
'num_words_emo': sum of the number of words + emoticons in each chat message
min_words:int
For algo3_0 ONLY
When filtering chunks to top users, at least how many words the top user should send
Output variables
- All algorithms will return a
result_json
, list of dictionaries in the format of{start:seconds, end:seconds}
whereseconds
is seconds elapsed since start of the stream. List is ordered from predicted best to worst timestamps. - All algorithms can save the returned list as a .json if
save_json=True
is passed in.
Background
Pillar is creating an innovative way to automatically select and splice clips from Twitch videos for streamers. This repo is focusing on the algorithm aspect. Three main algorithms are being tested.
Algorithms
- Algorithm 1: Find the best moments in clips based on where the most users participated. Most is defined as the ratio of unique users during a 2 min section to unique users for the entire session.
- Algorithm 2 Find the best moments in clips based on when rate of messages per user peaked. This involves answering the question "at which 2 min segment do the most users send the most messages?". If users X, Y, and Z all send 60% of their messages at timestamp range delta, then that timestamp might qualify as a "best moment"
- NOTE: Currently answers the question "at which 2 min segment do users send the most messages fastest"
- Algorithm 3 (WIP) Weigh each user by their chat rate, account age, etc. Heavier users predicted to chat more often at "best moment" timestamps
- STATUS: current weight determined by (
num_words_of_user
/num_words_of_top_user
) - Algorithm 3.5 Finds the best moments in clips based on most number of words/emojis/both used in chat
- STATUS: current weight determined by (
Timeit results
Results as of April 13, 2021 18:31 EST
run on big_df
with 80841 rows, 10 columns.
algo1 | algo2 | algo3_0 | algo3_5 |
---|---|---|---|
2.2 sec | 1 min 23 sec | 28.1 sec | 16.3 sec |
Build
To build and publish this package we are using the poetry python packager. It takes care of some background stuff that led to mistakes in the past.
Folder structure:
|-- dev_tools
|-- pillaralgos_dev
|-- __init__.py
|-- dev_helpers.py # aws connection, file retrieval script
|-- sanity_checks.py # placeholder
|-- README.md
|-- pyproject.toml
|-- prod
|-- pillaralgos # <---- note that poetry didn't require an additional subfolder
|-- helpers
|-- __init__.py
|-- data_handler.py
|-- emoji_getter.py
|-- __init__.py # must include version number
|-- algoXX.py # all algorithms in separate files
|-- brain.py
|-- LICENSE
|-- README.md
|-- pyproject.toml # must include version number
|-- reinstall_pill.sh # quick script to uninstall local pillaralgos, build and install new one
To publish just run the poetry publish --build
command after update version numbers as needed.
Changelog
- Moved
dev
andprod
folders out ofpypi
folder for ease of access - New licensing
- Added unit tests, sample data, sample answers (#6, #17, #18)
- Fixed use of iloc vs loc (#20)
- Algos2,3_0,3_5 now properly outputs correct lengths based on param input (#15, #21, #31)
- Algos return empty numpy array if no results found
- Created
brain.run()
that runs all algos and then returns common timestamps amongst them. Seehelp(brain)
andhelp(brain.run
for more info - Removed some old notebooks, moved some code to
archive.ipynb
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pillaralgos-1.0.17-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b1919facb2c5e790a7476ddf121998273c5dce5bd08da5f22dbe66289339dd20 |
|
MD5 | 557db60763ca1cbd08e1a9d25b66da13 |
|
BLAKE2b-256 | 2c57cdac83fe4907d64960661f341df359ed0813526181d418e6d389e6a0f870 |