sentibank

sentibank is an open database of expert-curated sentiment dictionaries, packaged as a Python library. It consolidates 15 original lexicons and 43 preprocessed dictionaries, spanning 7 genres and 6 domains, behind one loader, a bag-of-words analyser and a command-line interface. The resource is described in Oh (2024), published at ICWSM 2024.

TL;DR

Rule-based sentiment analysis still matters wherever a result has to be explained: policy, health, and most of computational social science. Its raw material, the expert-curated lexicon, is scattered across papers, personal web pages and dead links, in incompatible formats and with uneven documentation of how each was built. sentibank collects these lexicons in one place, keeps the original files next to preprocessed versions in a common format, and documents the provenance of each one.

One loader for every lexicon. archive.load().dict("VADER_v2014") returns a dictionary; the same call works for all 43 preprocessed identifiers, with caching for repeated loads.

Originals kept alongside. load.origin(...) returns the author's original file as a pandas DataFrame, so preprocessing choices can be checked and redone.

Documented provenance. Every lexicon has a page in the documentation covering composition, annotation methodology, evaluation and usage guidance, with references.

Install

Python 3.10 or later.

pip install sentibank

The spaCy model en_core_web_sm is downloaded on first use if it is missing.

Quickstart

from sentibank import archive
from sentibank.utils import analyze

load = archive.load()
vader = load.dict("VADER_v2014")          # preprocessed dictionary
vader_raw = load.origin("VADER_v2014")    # original file, as a pandas DataFrame

analyzer = analyze()
analyzer.sentiment("I am excited and happy about the new announcement!", dictionary="VADER_v2014")
# 4.1
analyzer.sentiment("We are pleased to announce record results to our shareholders.", dictionary="MASTER_v2022")
# {'Negative': 0, 'Uncertainty': 0, 'Constraining': 0, 'Positive': 1, 'Litigious': 0, 'Weak_Modal': 0, 'Strong_Modal': 0}
analyzer.dictionary("WordNet-Affect_v2006")   # summary of the lexicon's structure and scores

sentiment() is a bag-of-words analysis: term order is ignored. For score-based lexicons such as VADER_v2014 it sums the scores of matched terms and returns one number. For label-based lexicons such as MASTER_v2022 or GeneralInquirer_v2000 it counts matched terms per category and returns those counts.

The same operations are available from the command line:

sentibank list                                                   # identifiers of every dictionary
sentibank info VADER_v2014                                       # size, type, source
sentibank analyze VADER_v2014 "This product is absolutely amazing!"
sentibank analyze AFINN_v2015 --file reviews.txt --json          # a file, machine-readable output
sentibank export MASTER_v2022 --format csv --output master.csv  # csv or json

Other loader methods: list_available() returns the identifiers of dictionaries and originals, and clear_cache() drops cached dictionaries.

Dictionaries

Identifiers follow {NAME}_{VERSION} when only compulsory processing was applied to the base lexicon, and {NAME}_{VERSION}_{refinement} when a discretionary transformation was added. For example, NoVAD_v2013_boosted applies arousal-based adjustments that intensify extreme valence values and dampen neutral ones, giving a single richness-preserving score.

Repository structure

sentibank/
  archive.py       load: dict(), origin(), json_dict(), list_available(), clear_cache()
  utils.py         analyze: dictionary(), sentiment()
  validate.py      validation of contributed lexicons
  cli.py           sentibank list / info / analyze / export
  dict_arXiv/      one folder per lexicon: original CSV plus JSON and pickle versions
docs/              Jupyter Book: getting started, one page per lexicon, contribution guide
examples/          basic usage, dictionary comparison, CLI examples
tests/             pytest suite

Documentation

The documentation covers installation, loading and analysis, and has a page for each lexicon with its composition, methodology, evaluation and usage guidance.

Contributing

New expert-curated lexicons are welcome. The contribution criteria cover scope, composition, methodology, licensing and format. We are also working with domain experts on new gold-standard dictionaries for topics such as ideology, markets, cryptocurrency, politics and ESG; researchers who want to take part can write to research@socius.org.

Citation

@inproceedings{oh2024sentibank,
  title     = {sentibank: A Unified Resource of Sentiment Lexicons and Dictionaries},
  author    = {Oh, Nick},
  booktitle = {Proceedings of the International AAAI Conference on Web and Social Media},
  volume    = {18},
  pages     = {2003--2013},
  year      = {2024},
  doi       = {10.1609/icwsm.v18i1.31443}
}

License

The package is released under CC BY-NC-SA 4.0. Each lexicon keeps the license of its original authors; see docs/licenses.txt.