whispool 0.1.2

No project description provided

Whispool

Fast in memory word frequency count, with Rust

Installation

pip install whispool

Basic usage

We assume we have n sentences, with each sentence broken into words. And each sentence is

# each sentence could have diffrent length of words
sentence = [
            ['This', 'is', 'a', 'sentence'],
            ['This', 'is', 'another', 'sentence'],
            ['This', 'is', 'a', 'third', 'sentence', 'here'],
            ...
        ]

# each info list has to be IN THE SAME LENGTH
info = list([
            ['id1', 'category1', 'sci-fi', 'movie','Jan.'],
            ['id2', 'category1', 'romance', 'movie','Jan.'],
            ['id3', 'category2', 'sci-fi', 'tv','Feb.'],
            ...
        ])

Put in data

Now we put in data

from whispool import Whispool
from pathlib import Path

whisper_multi = Whispool(
        directory=Path("test_data"), threads=2, capacity=200)

whisper_multi.consume(sentence, info)
whisper_multi.consume(sentence_batch2, info_batch2)
...
whisper_multi.consume(sentence_batchN, info_batchN)

Fetch the result data

Now we want to see the hot key words of Sci-Fi movies in January.

df = whisper_multi.final_stats(
        [None, None, 'sci-fi', 'movie', 'Jan.'],
        top_n=2)

Build from rust

maturin build --release