NLP toolkit · annotator polarization research

Disagreement isn't noise.
It's where the pole is.

Average a room split between civil and toxic ratings and you get a "moderate" score that describes no one in the room. Polartox finds the socio-demographic split that explains the disagreement instead of erasing it — and does it recursively, subgroup by subgroup.

View on GitHub PyPI package

Illustrative — a single split from tree.render() on synthetic data. Bars are ratings 1–5.

The idea

An annotation dataset's average rating is a fact about the average annotator — a person who does not exist.

Annotator disagreement on toxicity, bias, and safety ratings is usually treated as noise to smooth over: majority vote, mean rating, Krippendorff's alpha as a single number. That discards exactly the signal that matters when the disagreement isn't random — when it tracks who is rating, not just what they're rating.

Polartox measures that structure directly. nDFU (normalized Distance From Unimodality) scores how bimodal a rating distribution is. Polarized Trees then recursively splits annotators by socio-demographic dimension — gender, politics, age, whatever the study collected — searching for the split that explains the most of that bimodality, until each remaining subgroup agrees with itself.

The result isn't "this text is polarizing." It's which intersectional subgroups land on the civil pole, which land on the toxic pole, and how consistently that holds across a whole corpus.

Approach

One text, five steps, repeated recursively per subgroup.

Filter

Compute nDFU per text; keep only those at or above theta_filter. A text where everyone already agrees has nothing for the method to explain.

Split

Score every remaining candidate dimension by PEG (Polarization Explanation Gain) and take the one that explains the most bimodality. PEG can be measured against the most or least polarized subgroup, their size-weighted average, or a mean or harmonic combination of the three (variant).

Stop

Recursion halts on a subgroup once it's too small (min_size), already agrees (theta_stop), the best split isn't worth it (h), or the tree hits max_depth.

Label

Each leaf's ratings are called toxic above theta_pole; the leaf itself is labelled by majority vote — toxic, civil, or indeterminate on a tie.

Summarize

Across every tree in the corpus: F (which dimensions split most), C (how consistently a subgroup lands on one pole), P (how much polarization each split explains).

Five modules

Generate the data you can validate against, build one tree or a corpus of them, tune it, then compare the PEG formulations.

polartox.datagen

stable

Synthetic annotator pools with known ground truth — each text gets k active demographic dimensions with a real toxic/civil lean and intensity, so recovery can be checked against an actual answer.

# build a pool, generate 100 texts with known active dims
from polartox.datagen import (
    AnnotatorPool, DEFAULT_DIMENSIONS,
    DEFAULT_DEPTH_WEIGHTS, DEFAULT_INTENSITY_RANGE,
)

pool = AnnotatorPool(
    dimensions=DEFAULT_DIMENSIONS, scale=5,
    intensity_range=DEFAULT_INTENSITY_RANGE,
    depth_weights=DEFAULT_DEPTH_WEIGHTS,
    annotators_per_identity=10,
)
dataset, ground_truth = pool.generate_dataset(
    n_texts=100, noise=0.05, seed=42,
)

polartox.polarized_tree

stable

The single-tree building block. Build and inspect one text's tree directly — no corpus, no pipeline — and query its structure through a small API (render, inspect, find_node, ...).

# build one tree, print it, walk its nodes
from polartox.polarized_tree import PolarizedTree

tree = PolarizedTree.build(
    text_data, dims=["gender","age"], scale=5,
    min_size=10, h=0.05, max_depth=4,
)
tree.render()
tree.inspect(dataset, show_distributions=True)

polartox.pipeline

stable

Corpus-level orchestration: filters a whole dataset down to polarized texts, builds a PolarizedTree per text, and aggregates F/C/P plus ground-truth-free diagnostics across all of them.

# run the full method over a corpus
from polartox.pipeline import PolarizedTreesPipeline

pipe = PolarizedTreesPipeline(
    dims=["gender","politics","age"],
    scale=5, theta_filter=0.3, h=0.15, max_depth=8,
)
results = pipe.run_full_evaluation(dataset)
results["F"], results["C"], results["P"]

polartox.benchmark

stable

Systematic hyperparameter search over the pipeline's own config space, ranked by recovery metrics against known ground truth — the paper's default space alone has 3,240 valid configurations.

# search, rank, and hand back the winning pipeline
from polartox.benchmark import PolarizedTreesBenchmark

bench = PolarizedTreesBenchmark(
    pipeline=pipe, annotations=dataset, ground_truth=ground_truth,
    strategy="random", n_runs=100, selection_metric="jaccard",
)
bench.run()
best = bench.get_best_pipeline()

polartox.peg_comparison

stable

The last step of the chain: take the benchmark's best settings, hold everything fixed except the PEG formulation (max, weighted, min, mean, harmonic), and compare. What do its trees look like (leaves, depth, group size, the dimensions used), how close are the trees of two formulations (adjusted Rand index, normalized mutual information, same dimensions), and with ground truth, which formulation recovers the true dimensions? Ground truth is optional, so it works on real data too.

# the best settings, one formulation at a time
from polartox import PEGComparison

cmp = PEGComparison.from_benchmark(bench)   # corpora, dims, scale
cmp.run(bench.top_configs_.head(3))
cmp.overview()       # recovery and tree shape per formulation
cmp.ari_matrix()     # do they split annotators the same way?
cmp.disagreement()   # the texts on which they differ most

What powers a split — nDFU & friends

One number for "does this group agree with itself?"

0.04

unimodal — the group agrees

0.94

bimodal — the group is split

nDFU (normalized Distance From Unimodality) is computed straight from the rating histogram, no annotator identity needed: near 0 means one hump, agreement; near 1 means two humps at opposite ends of the scale — the signature of a genuine pole. Polartox depends on the collaborative ndfu package for the score itself rather than reimplementing it.

from ndfu import dfu, pdf

hist = pdf(ratings, range(1, scale + 1))
score = dfu(hist)  # 0 = agreement, ~1 = split

PEG (Polarization Explanation Gain) then measures how much of that bimodality a candidate split would explain if taken, and theta_pole sets the rating cutoff a leaf's individual ratings must clear to count toward its toxic majority vote.

From the paper's synthetic benchmark

Validated where the true answer is known, before it's trusted on real data.

3,240

valid configurations in the default search space

800

randomly sampled & ranked by mean Jaccard

0.89

mean Jaccard of the best configuration, over the three benchmark corpora

0–4

active dimensions per synthetic text, by design

Paper, notebooks & docs

Everything above is runnable, not illustrative.

The full methodology — PEG variants, stopping criteria, and the model-selection procedure — is written up in the accompanying paper. Every module has a runnable notebook, including an end-to-end run on real conversational-safety annotations (DICES-350/990) with no synthetic ground truth in sight.

Read the paper (PDF) Notebook — synthetic data generation Notebook — one tree, directly Notebook — full pipeline & F/C/P Notebook — hyperparameter benchmark Notebook — comparing the PEG formulations Notebook — exploring the benchmark results Notebook — PEG comparison on the benchmark Notebook — real-data inference (DICES)