scweatwords 1.0.0

Calculate SC-WEAT bias from GloVe 200d Gigaword embeddings.

Gender Bias in Word Embeddings

A command-line tool for measuring gender bias in GloVe word embeddings using the Single-Category Word Embedding Association Test (SC-WEAT).

Student: Katharina Nemtsev

ID Number: 19-922-137

Repo: gitlab

Background

Word embeddings are reflective of the texts they were trained on. Much research has shown that there is a gender bias reflected in word embeddings. With this module, I want to make it possible to quickly find the gender bias of words based on the newest GloVe word embeddings.

The GloVe corpus used is the 2024 Wikipedia + Gigaword 5 (11.9B tokens, 1.2M vocab, uncased, 200d vectors, 1.1 GB download), downloadable also at: GloVe. The program already includes the necessary file, so no need to download anything for you.

The SC-WEAT methodology is based on Caliskan et al. (2022), Gender Bias in Word Embeddings: A Comprehensive Analysis of Frequency, Syntax, and Semantics arXiv, which used the SC-WEAT score to demonstrate the widespread prevalence of gender bias across GloVe and fastText embeddings. I also used their anchor words, which came from a previous paper of Caliskan et al. (2017): arXiv. I am not very familiar with the SC-WEAT measurement and only learned about it for this project, so please excuse any comprehension errors you may find.

SC-WEAT Formula

For a given input word w, the module computes a Cohen's d effect size by comparing how close w is to a set of male anchor words versus a set of female anchor words in vector space:

ES(w) = ( mean_cos(w, male_anchors) - mean_cos(w, female_anchors) )
        ─────────────────────────────────────────────────────────────
                    std_dev( all cosine similarities )

A negative effect size indicates male association; a positive effect size indicates female association. The magnitude follows Cohen's d conventions, as also described by Caliskan et al.:

| |d| | Interpretation | |-----------|----------------| | < 0.20 | Negligible | | 0.20–0.49 | Small | | 0.50–0.79 | Medium | | ≥ 0.80 | Large |

Each side requires at least 8 anchor words, as specified by the SC-WEAT methodology. The default anchors can be changed by editing anchors.json directly.

Setup

Requirements

• Python 3.13
• numpy

Usage

Installation:

python3 pip install scweatwords

Score a single word

python3 embeddings.py --word nurse
python3 embeddings.py --word engineer

Score a list of words from a file

python3 embeddings.py --list words.txt

One word per line. Words not found in the GloVe vocabulary are skipped with an error.

Find nearest neighbours

Add --neighbours N to either the word or file to retrieve the N most similar words in the embedding space, alongside their similarity to the input word:

python3 embeddings.py --word doctor --neighbours 10
python3 embeddings.py --list jobs.txt --neighbours 5

Be aware: In the list option, this iterates over the entire GloVe file once for each word. The size of N will not have a noticeable impact on your runtime, but the amount of words in your list will.

Anchor words

The gender anchor words are stored in anchors.json. The defaults are:

{
  "male":   ["male", "he", "him", "his", "man", "boy", "son", "brother"],
  "female": ["female", "she", "her", "hers", "woman", "girl", "daughter", "sister"]
}

These are taken from the two Caliskan et al. papers mentioned earlier.

Reference

Caliskan, A. et al. (2022) Gender bias in word embeddings: A comprehensive analysis of frequency, syntax, and semantics, arXiv.org. Available at: https://doi.org/10.48550/arXiv.2206.03390 (Accessed: 07 June 2026).

Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. [pdf] [bib]

Riley Carlson, John Bauer, and Christopher D. Manning. 2025. A New Pair of GloVes. [pdf]