nltk-vader-sentiment

A modern, dependency-free reimplementation of the VADER sentiment analyzer (Hutto & Gilbert, ICWSM-14) — with the sentiment and emoji lexicons bundled as Python data. No downloads, no pickles, no file parsing at runtime, no dependencies, and no nltk import anywhere.

pip install nltk-vader-sentiment

>>> from vader import polarity_scores
>>> polarity_scores("VADER is smart, handsome, and funny!")
{'neg': 0.0, 'neu': 0.248, 'pos': 0.752, 'compound': 0.8439}

>>> from vader import SentimentIntensityAnalyzer
>>> analyzer = SentimentIntensityAnalyzer()          # reuse this; it is thread-safe
>>> analyzer.polarity_scores("Not bad at all :) ❤️")
{'neg': 0.0, 'neu': 0.285, 'pos': 0.715, 'compound': 0.8765}

There is also a CLI:

$ vader "This is amazingly good!"
{"neg": 0.0, "neu": 0.463, "pos": 0.537, "compound": 0.54}
$ vader < reviews.txt        # one JSON object per line

Why this package

The two maintained homes of VADER have both drifted and stalled:

• cjhutto/vaderSentiment — the original. Its last PyPI release (3.3.2) is from 2020, the repository is essentially dormant, it installs requests for a demo block, and its issue tracker documents years of unfixed scoring bugs.
• nltk.sentiment.vader — a 2016-era snapshot that has drifted from upstream (nltk#2592): no emoji handling, a different tokenizer that misses (super) and word!! (nltk#3071), and a vader_lexicon that must be fetched with nltk.download() at runtime.

This package reimplements the algorithm from scratch against the upstream reference, fixes the catalogued defects (each fix carries a regression test naming its issue), and ships everything needed at pip install time. It is built to the same standard as nltk-punkt-tokenize, so that NLTK could one day vendor sentiment analysis out to it — which is why nothing here may ever import nltk (enforced by tests three ways).

Parity with upstream

Correctness is measured, not claimed. Against a pinned snapshot of upstream vaderSentiment on 1,219 cases (the upstream demo sentences plus samples from all four ICWSM-14 ground-truth datasets):

• 1,189/1,219 (97.5%) identical compound scores. Every one of the 30 divergences is traced to a deliberate bug fix (see below).
• Correlation with the ICWSM-14 human ground truth matches upstream to within ±0.0003 on all four datasets:

Fixed defects

Engine fixes (relative to upstream master and/or NLTK):

Lexicon fixes (resolved at build time by tools/build_data.py, every resolution logged):

• 14 duplicate keys with conflicting scores → upstream's effective (last-wins) value kept for parity (vaderSentiment#122); the two sign-flipped emoticons d:/d= additionally get the mean over both rating rounds via the curated layer.
• 14 uppercase keys were unreachable (lookups lowercase the token): keys are case-folded, keeping the previously-reachable value on collision; :Þ and :-Þ, which had no lowercase twin and were dead data, now work.
• Curated additions (details): kindly → 0.0 (vaderSentiment#155); the slaughter* family, absent while killed/murdered score −3.5/−3.4 (nltk#3507). Disable all curated changes with SentimentIntensityAnalyzer(curated=False).

Customisation

Every word list is per-instance constructor data — no module-global mutation (vaderSentiment#102, #105, #111):

analyzer = SentimentIntensityAnalyzer(
    extra_lexicon={"rugpull": -2.9},   # domain terms on top of the default
    negations=[...],                   # replace the negation list
    booster={...},                     # replace degree adverbs
    special_cases={...},               # phrase overrides
    idioms={},                         # {} disables sentiment-laden idioms
    alpha=15.0,                        # compound normalisation constant
)

Instances are immutable after construction and safe to share across threads.

Design and security

Same rules as nltk-punkt-tokenize:

• Nothing imports nltk — not the library, not the tests. Enforced by AST scan, by a child interpreter where import nltk raises, and by an empty dependency list. The intended dependency arrow is nltk → nltk-vader-sentiment, never a cycle back.
• Zero dependencies, stdlib only. The upstream package installs requests for a demo; this one installs nothing.
• Lexicons are code-like data: generated Python literal modules under vader/data/, imported, never parsed. The generator re-parses its own output and rejects any AST node that is not literal data.
• No file parser ships: tools/ (the only code that reads lexicon files) is excluded from both the wheel and the sdist. Get it from the repository, deliberately.
• No pickles, no downloader, no network, no eval, no runtime file IO — all test-enforced.

Known limitations (documented, not "fixed")

Inherent to a lexicon-and-rules model — if these matter, use a trained model:

• Sarcasm/irony: "The movie was funny... not" stays positive.
• Compound saturation: compound is sum/√(sum²+α), so long texts saturate towards ±1 and scores are not comparable across lengths (vaderSentiment#151). Score per sentence and aggregate.
• Domain sensitivity: tuned for social media; weaker on finance, health and news (SentiBench; use extra_lexicon for domain terms).
• English only. Do not feed it other languages or machine translations.
• Lexicon bias: published audits find disability-related terms skew negative and queer identity terms are absent (Ungless et al. 2023; TrustNLP 2023). The curated layer fixes only clear-cut cases; audit against your own data before making decisions about people.

Development

pip install -e ".[dev]"
pytest                  # package suite (runs without tools/, as in an sdist)
pytest tools/tests      # maintainer suite for the data converters
ruff check src tests tools && mypy

Regenerate the data modules from an upstream checkout:

python tools/build_data.py /path/to/vaderSentiment/vaderSentiment
python tools/generate_parity_fixtures.py /path/to/vaderSentiment

Citation and license

Licensed under the Apache License 2.0 (see LICENSE). The VADER algorithm and lexicons are by C.J. Hutto, MIT-licensed (see NOTICE). If you use VADER, please cite:

Hutto, C.J. & Gilbert, E.E. (2014). VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text. Eighth International Conference on Weblogs and Social Media (ICWSM-14). Ann Arbor, MI, June 2014.