polartox

NLP toolkit for annotator polarization research. Provides tools for synthetic dataset generation and polarization detection in human annotation studies.

Install

pip install polartox

Tools

polartox.datagen

Builds a pool of annotators with explicit demographic identities and generates annotation datasets where every text independently gets k active dimensions that drive its disagreement:

• k = 0 — no demographic dimension explains the disagreement, providing a true unimodal negative control
• k ≥ 1 — a subset of dimensions is active, each with its own random toxic/civil lean split and a continuous intensity (alpha) controlling how strongly it pulls toward its pole

Identities' rating distributions are built by taking the elementwise product of their active-dimension shapes — signal composes rather than averages away, allowing the generated data to span the full nDFU range.

from polartox.datagen import AnnotatorPool, DEFAULT_DIMENSIONS, DEFAULT_DEPTH_WEIGHTS, DEFAULT_INTENSITY_RANGE

pool = AnnotatorPool(
    dimensions=DEFAULT_DIMENSIONS,
    scale=5,
    intensity_range=DEFAULT_INTENSITY_RANGE,
    depth_weights=DEFAULT_DEPTH_WEIGHTS,
    annotators_per_identity=10,
)

result = pool.generate_dataset(
    n_texts=100,
    n_annotators_per_text=150,
    noise=0.05,
    seed=42,
)
dataset, ground_truth = result
# dataset columns: text_id, annotator_id, <dimensions>, rating
# ground_truth: per-text active_dims, lean (toxic/civil split), and alpha (intensity)

nDFU scoring is provided by the collaborative ndfu package (Pavlopoulos & Likas, 2024) rather than reimplemented here:

from ndfu import dfu, pdf

text_data = dataset[dataset["text_id"] == 0]
hist = pdf(text_data["rating"].tolist(), range(1, pool.scale + 1))
score = dfu(hist)

polartox.polarized_trees

Given an annotation dataset, recursively partitions annotators by demographic dimension to find the specific subgroups most polarized on a given text — which dimensions drive disagreement, and which intersectional subgroups diverge most strongly.

from polartox.polarized_trees import PolarizedTreesPipeline

pipe = PolarizedTreesPipeline(
    dims=list(DEFAULT_DIMENSIONS.keys()),
    scale=5,
    theta_filter=0.3,      # minimum nDFU required for analysis
    min_size_frac=0.03,    # minimum subgroup size
    h=0.15,                # relative PRG threshold
    max_depth=6,
    relative_h=True
)

results = pipe.run_full_evaluation(dataset)
# results["F"], results["C"], results["P"] -- dataset-level summaries
# results["diagnostics"] -- ground-truth-free corpus diagnostics

# on synthetic data, pass ground_truth=... for recovery metrics
results = pipe.run_full_evaluation(dataset, ground_truth=ground_truth)
results["recovery"]   # per-text Jaccard / precision / recall / exact match

Full API documentation and demos are available on GitHub.

Benchmark configuration selection, hyperparameter results, synthetic recovery results, and unseen-corpus evaluation are documented in benchmarks/README.md.

For the full methodology and paper-specific details, see polarized_trees.pdf.

Changelog

See the CHANGELOG for release history.