hct-bootstrap 0.2.0

A package using the Higher Criticism statistic to find discriminating words in two text corpora.

HCT-Bootstrap: Discriminating Words in Text Corpora

A Python package for identifying stable, statistically discriminating words between two text corpora, using Higher Criticism thresholding with bootstrap stabilization and an empirical within-corpus null.

Overview

This package provides tools to perform a robust comparison of two texts or two collections of texts. The goal is to move beyond simple word frequency counts and identify words that are "surprisingly" frequent in one corpus relative to the other — a hallmark of stylistic difference.

Two problems make this harder than it looks. First, a vocabulary contains thousands of candidate words, so some will appear discriminating by chance alone. Second, a single statistical run is unstable: small perturbations of the data can change which words are selected. This package addresses both. It repeats the selection across bootstrap resamples and keeps only the words that survive every one, and it calibrates how many resamples are needed by running the same procedure on a null comparison built from corpora that are known to share a generating mechanism.

The result is a word list produced without any analyst-chosen frequency cutoff or threshold, together with a false-discovery bound.

The package is particularly useful for analyzing and comparing text generated by Large Language Models, comparing authors, or tracking stylistic change in a corpus over time.

Higher Criticism for Word-Frequency Tables

The core statistical engine is the Higher Criticism (HC) test, adapted for textual analysis as described in Kipnis (2022).

Comparing two corpora involves thousands of individual hypothesis tests — one for each unique word. For each word we can compute a p-value representing the probability of observing its frequency difference by chance. But with thousands of tests, many small p-values will occur randomly: the multiple testing problem.

Traditional corrections like Bonferroni are too conservative here, often missing subtle but collectively significant signals. The Higher Criticism statistic is instead designed to detect a sparse, heterogeneous signal — which matches the assumption that an author's or model's style is defined not by a shift in every word, but by a small, unknown subset of words used idiosyncratically.

HC examines the ordered sequence of all p-values and finds the threshold where the number of surprisingly small p-values is most significant relative to what the null hypothesis would produce. This makes it well suited to identifying a faint but consistent stylistic fingerprint. The core statistical code is adapted from Alon Kipnis's TwoSampleHC repository.

References

• Kipnis, A. (2022). Higher Criticism for Discriminating Word-Frequency Tables and Authorship Attribution. The Annals of Applied Statistics, 16(2), 1236–1252. https://arxiv.org/pdf/1911.01208

• Donoho, D., & Jin, J. (2004). Higher Criticism for detecting sparse heterogeneous mixtures. The Annals of Statistics, 32(3), 962–994. https://arxiv.org/pdf/math/0410072

Installation

1. Activate your virtual environment. Python 3.8 or later.

2. Install from PyPI:

   pip install hct-bootstrap
   

3. Download the spaCy model. The package uses spaCy for tokenization, and the English model is not installed automatically:

   python -m spacy download en_core_web_sm
   

4. Optional display extras. For the in-notebook results table and the diagnostic plot:

   pip install "hct-bootstrap[display]"
   

How the Method Works

The procedure has three moving parts, and two of them are the arguments you will most often set.

Bootstrap stabilization. At step n, the package has drawn n bootstrap resamples of the two corpora and run HC thresholding on each. A word counts as surviving only if it was selected in all n runs. The surviving count N̂(n) therefore only shrinks as n grows.

The null comparison. In parallel, the same procedure runs on a comparison where the two sides are known to come from the same source, so every surviving word is by construction a false discovery. N̂₀(n) is that count, taken as the maximum across the available null comparisons (conservative). Set it with null_method:

null_method	Null construction
'split' (default)	Each input corpus is shuffled and split into two non-overlapping halves, which are compared to each other.
'resample'	Two independent with-replacement resamples are drawn from each input corpus and compared.
'user'	You supply a same-source corpus for each input via null_corpus1_docs / null_corpus2_docs; each original corpus is compared against its counterpart.

The stopping rule. How many bootstrap iterations to run is decided by the data, not by you. Set it with q:

q	Rule
None (default)	Strict: stop at the first n where no null word survives (N̂₀(n) = 0).
a float in (0, 1]	FDR: stop at the first n where FDP⁺(n) = (1 + N̂₀(n)) / max(N̂(n), 1) falls to q. The selected set then has false-discovery rate at most q.

The final selection is the set of words surviving all n iterations, intersected with the words selected by a single HC run on the full corpora.

Quick Start

This example builds two synthetic corpora that differ only in a small set of injected "AI style" words, then recovers them.

from hct_bootstrap import analyze_and_display
import random

# Set random seed for reproducibility
random.seed(42)

background_vocab = [
    "analysis", "system", "data", "model", "research", "method", "result", "time", "user", "network",
    "process", "study", "group", "problem", "level", "information", "effect", "change", "design", "feature",
    "performance", "approach", "case", "point", "development", "variable", "pattern", "value", "algorithm", "test",
    "environment", "structure", "condition", "behavior", "factor", "application", "control", "theory", "practice", "solution",
    "science", "technology", "paper", "review", "framework", "function", "state", "rule", "context", "form",
    "concept", "equation", "experiment", "measure", "source", "type", "area", "base", "number", "size",
    "quality", "property", "task", "technique", "principle", "resource", "material", "energy", "rate", "scale",
    "phase", "stage", "component", "class", "architecture", "mechanism", "relation", "role", "impact", "access"
]

ai_style_words = ["delve", "intricate", "underscore", "meticulous", "crucial",
                  "tapestry", "robust", "realm", "nuance", "leverage"]

human_corpus = []
ai_corpus = []

# 1. Create the 'Human' corpus: 1000 documents of 250 random background words
for _ in range(1000):
    doc = " ".join(random.choices(background_vocab, k=250))
    human_corpus.append(doc)

# 2. Create the 'AI' corpus: same length and vocabulary, plus the injected style words
for _ in range(1000):
    doc = " ".join(random.choices(background_vocab, k=250))
    doc += " " + " ".join(ai_style_words)
    ai_corpus.append(doc)

# 3. Run the analysis and display the report.
#    coupled=False because the two lists are not paired one-to-one. If you had
#    generated both corpora from a shared list of prompts, so that
#    corpus1_docs[i] and corpus2_docs[i] come from the same prompt, you would
#    set coupled=True (the lists must then be of equal length).
#    q=None uses the strict stopping rule; null_method='split' builds the null
#    by halving each input corpus.
results = analyze_and_display(
    corpus1_docs=human_corpus,
    corpus1_name="Human",
    corpus2_docs=ai_corpus,
    corpus2_name="AI",
    coupled=False,
    q=None,
    null_method="split",
)

Example Report Output

=== BS-HCT Analysis Report ===

Comparing: 'Human' vs. 'AI'
Coupled test comparison: False
Null construction: split (always uncoupled)
Stopping rule: strict (stop when no null word survives)
Max t: 1000
--------------------
  t=    1 | N_test=   17 | N0_max=  12 | FDP+=0.7647
  t=    2 | N_test=   12 | N0_max=   6 | FDP+=0.5833
  t=    3 | N_test=   11 | N0_max=   3 | FDP+=0.3636
  t=    4 | N_test=   11 | N0_max=   2 | FDP+=0.2727
  t=    5 | N_test=   11 | N0_max=   2 | FDP+=0.2727
  t=    6 | N_test=   11 | N0_max=   2 | FDP+=0.2727
  t=    7 | N_test=   11 | N0_max=   2 | FDP+=0.2727
  t=    8 | N_test=   11 | N0_max=   2 | FDP+=0.2727
  t=    9 | N_test=   11 | N0_max=   2 | FDP+=0.2727
  t=   10 | N_test=   11 | N0_max=   1 | FDP+=0.1818
  t=   11 | N_test=   11 | N0_max=   1 | FDP+=0.1818
  t=   12 | N_test=   11 | N0_max=   1 | FDP+=0.1818
  t=   13 | N_test=   11 | N0_max=   0 | FDP+=0.0909

--- Corpus Information (Post-Cleaning) ---
# words in Human: 249849
# words in AI: 259881

--- Full-Corpus HCT ---
HCT candidate words: 17
Full-corpus HC score: 4.6882

--- Stopping ---
Rule fired at iteration n = 13 (ran 13 iterations).

--- Stable Discriminating Words (11) ---

More frequent in 'Human':
principle

More frequent in 'AI':
crucial, delve, intricate, leverage, meticulous, nuance, realm, robust, tapestry, underscore

Selected words with HCT p-values:

word	p_value	more_frequent_in
nuance	4.003404e-271	AI
crucial	4.003404e-271	AI
delve	4.003404e-271	AI
intricate	4.003404e-271	AI
leverage	4.003404e-271	AI
tapestry	4.003404e-271	AI
robust	4.003404e-271	AI
meticulous	4.003404e-271	AI
underscore	4.003404e-271	AI
realm	4.003404e-271	AI
principle	1.631736e-07	Human

=== End of Report ===

A plot of N̂(n), N̂₀(n), and FDP⁺(n) against n is displayed after the report when matplotlib is installed.

Reading the report

The per-iteration trace shows the mechanism directly. A single HC run on the full corpora flags 17 candidate words, but stability winnows that to 11 by the third resample. Meanwhile the null count falls from 12 to 0, and the run stops at n = 13, the first iteration where no null word survives.

All ten injected style words are recovered. The eleventh selection, principle, is a chance artifact: both corpora draw from the same background vocabulary, so any difference in a background word is noise. Notice that its p-value is 264 orders of magnitude larger than the others — it sits exactly at the boundary of what the stability requirement can defend. This is what the reported FDP⁺ of 0.0909 means in practice: one false discovery among eleven selections, matching the bound the procedure derived for itself before anyone looked at the words.

To silence the per-iteration trace, pass verbose=False.

API Reference

The package exposes two functions. They take identical arguments; analyze_and_display prints the formatted report and plot shown above, while discriminate returns the results silently for programmatic use.

analyze_and_display(...) and discriminate(...)

Required

• corpus1_docs, corpus2_docs (List[str]): the documents of each corpus.
• corpus1_name, corpus2_name (str): labels used throughout the report and results.
• coupled (bool): True if corpus1_docs[i] is paired with corpus2_docs[i] — for example, both generated from the same prompt. Requires lists of equal length, and resampling then draws the same document indices for both corpora, preserving the pairing. False resamples each corpus independently.

Selection behavior

• q (float, optional, default None): the stopping rule. None for the strict rule, or a value in (0, 1] for the FDR rule. See How the Method Works above.
• null_method (str, default 'split'): 'split', 'resample', or 'user'.
• null_corpus1_docs, null_corpus2_docs (List[str], optional): required when null_method='user'. One same-source null corpus per input corpus.
• null_coupled (bool or (bool, bool), optional): whether each user-supplied null corpus is prompt-paired with its test corpus, in which case the pair is cleaned jointly to preserve alignment. Pass a single boolean to apply to both null comparisons, or a two-element tuple to set them separately. Only meaningful with null_method='user'; defaults to matching coupled.
• max_t (int, default 1000): safety cap on bootstrap iterations if the stopping rule never fires.
• random_seed (int, default 42): seed for reproducible resampling.

Pre-processing

• pos_tags (List[str], optional, default None): restrict the analysis to these spaCy part-of-speech tags, for example ['NOUN', 'VERB']. None keeps all parts of speech. Named entities, punctuation, and tokens mixing letters with digits are always removed, and hyphenated compounds such as real-world are kept intact.
• nlp (spacy.Language, optional): a pre-loaded spaCy pipeline. If None, default_spacy_model is loaded internally and cached.
• default_spacy_model (str, default 'en_core_web_sm'): the model to load when nlp is not supplied.

Output

• verbose (bool, default True): print the per-iteration trace of N̂(n), N̂₀(n), and FDP⁺(n).

Return value

Both functions return a dictionary:

Key	Contents
selected_words	Sorted list of the stable discriminating words.
selected_words_df	Those words with p_value and more_frequent_in, ascending by p-value.
n_rule, t_stop	The iteration at which the rule fired, or None. Both keys hold the same value.
fdp_at_stop	FDP⁺(n) at the stopping iteration.
rule	'strict' or 'fdr'.
q, null_method	The settings used, echoed back.
iterations_run	Number of iterations actually run.
empty_reason	Explanation if the selection came back empty, otherwise None.
fdp_curve	List of (n, N̂(n), N̂₀(n), FDP⁺(n)) tuples — the data behind the plot.
full_data_hc_score	HC score from the single run on the full corpora.
full_data_results_df	Words selected by that single full-corpus run.
test_survivors_df	Surviving test words at the stopping iteration.
null_survivors_dfs	Surviving null words, one DataFrame per null comparison.
cleaned_corpus1, cleaned_corpus2	The cleaned corpora as joined strings.

Data

The corpora generated for the accompanying study — LLM-written scientific abstracts across sampling temperatures and model families, human versus LLM legislative summaries, and base versus instruction-tuned model summaries — are available in the corpora/ directory of this repository.