polartox

NLP toolkit for annotator polarization research. Provides tools for synthetic dataset generation and polarization detection in human annotation studies.

Install

pip install polartox

Tools

polartox.datagen

Builds a pool of annotators with explicit demographic identities and generates annotation datasets where every text independently gets k active dimensions (0–4) that drive its disagreement:

• k = 0 — no dimension explains anything, a true unimodal negative control
• k ≥ 1 — a random subset of dimensions is active, each with its own random toxic/civil lean split and a continuous intensity (alpha) controlling how strongly it pulls toward its pole

Identities' rating distributions are built by taking the elementwise product of their active-dimension shapes — signal composes rather than averages away, reaching the full nDFU range instead of collapsing toward the middle.

from polartox.datagen import AnnotatorPool, DEFAULT_DIMENSIONS, DEFAULT_DEPTH_WEIGHTS, DEFAULT_INTENSITY_RANGE

pool = AnnotatorPool(
    dimensions=DEFAULT_DIMENSIONS,
    scale=5,
    intensity_range=DEFAULT_INTENSITY_RANGE,
    depth_weights=DEFAULT_DEPTH_WEIGHTS,
    annotators_per_identity=10,
)

result = pool.generate_dataset(
    n_texts=100,
    n_annotators_per_text=150,
    noise=0.05,
    seed=42,
)
dataset, ground_truth = result
# dataset columns: text_id, annotator_id, <dimensions>, rating
# ground_truth: per-text active_dims, lean (toxic/civil split), and alpha (intensity)

nDFU scoring is provided by the collaborative ndfu package (Pavlopoulos & Likas, 2024) rather than reimplemented here:

from ndfu import dfu, pdf

text_data = dataset[dataset["text_id"] == 0]
hist = pdf(text_data["rating"].tolist(), range(1, pool.scale + 1))
score = dfu(hist)

polartox.polarized_trees

Given an annotation dataset, recursively partitions annotators by demographic dimension to find the specific subgroups most polarized on a given text — which dimensions drive disagreement, and which intersectional subgroups diverge most strongly.

from polartox.polarized_trees import PolarizedTreesPipeline

pipe = PolarizedTreesPipeline(
    dims=list(DEFAULT_DIMENSIONS.keys()),
    scale=5,
    theta_filter=0.3,      # how polarized a text must be to analyze at all
    min_size_frac=0.03,    # min subgroup size, as a fraction of each text's annotators
    h=0.05,                # PRG gain threshold
    max_depth=6,
    relative_h=True
)

results = pipe.run_full_evaluation(dataset)
# results["F"], results["C"], results["P"] -- dataset-level metrics
# results["diagnostics"] -- ground-truth-free corpus stats, usable on real data

# on synthetic data, pass ground_truth=... (dict per text_id) for recovery metrics
results = pipe.run_full_evaluation(dataset, ground_truth=ground_truth)
results["recovery"]   # per-text jaccard / precision / recall / exact_match

Full API documentation available on GitHub.

Changelog

See CHANGELOG.md for release history.