wordshift 0.1.0

Fast weighted-average word shift computation with bundled labMT lexicons, powered by Rust

wordshift

Fast weighted-average word shift computation with bundled labMT lexicons, powered by Rust (wordshift-core).

A word shift quantifies which words drive the difference in average sentiment between two texts, and in which direction. This is a lean Rust port of Shifterator's WeightedAvgShift for the single-lexicon (sentiment) case.

Install

pip install wordshift        # once published
# or, from source:
maturin build --release -m crates/wordshift-py/Cargo.toml && pip install target/wheels/wordshift-*.whl

Usage

import wordshift

type2freq_1 = {"happy": 1, "sad": 9, "love": 2}
type2freq_2 = {"happy": 9, "sad": 1, "love": 8}

result = wordshift.weighted_avg_shift(
    type2freq_1,
    type2freq_2,
    lexicon="labMT_English",   # or "English", "French", ...
    reference_value=None,       # None -> system 1's mean score (the baseline)
    top_n=50,                   # cap returned entries (0 = all)
)

for entry in result["entries"][:5]:
    print(entry["type"], round(entry["shift_score"], 4))

result is a dict with:

• entries — per-word {type, p_diff, s_diff, p_avg, s_ref_diff, shift_score}, sorted by absolute contribution. shift_score is normalized so absolute scores sum to 1 (the "variation" scheme).
• component_sums — the six sign-quadrant totals for the stacked summary bars.
• total_diff, norm, s_avg_1, s_avg_2, reference_value, normalization.

To supply your own scores instead of labMT, use weighted_avg_shift_custom:

scores = {"happy": 8.0, "sad": 2.0}
result = wordshift.weighted_avg_shift_custom(type2freq_1, type2freq_2, scores)

Words present in the texts but missing from the lexicon are excluded.

Reference value

By convention, system 1 is the baseline and system 2 is read as a shift away from it. reference_value=None (or "average") partitions words around system 1's frequency-weighted mean score. Pass a float for a fixed reference — labMT's neutral midpoint is 5.0.