hct-bootstrap 0.1.0

A package using the Higher Criticism statistic to find discriminating words in two texts.

HCT-Bootstrap: Discriminating Words in Text Corpora

A Python package for identifying statistically significant, discriminating words between two text corpora using the Higher Criticism method and bootstrapping for stability analysis.

Overview

This package provides tools to perform a robust stylistic comparison of two texts or two collections of texts. The primary goal is to move beyond simple word frequency counts and identify words that are "surprisingly" frequent in one corpus compared to the other, a hallmark of stylistic difference.

This is particularly useful for analyzing and comparing text generated by Large Language Models (LLMs), comparing different authors, or tracking stylistic changes in a corpus over time. The package provides both a core computational function for raw data analysis and a convenient all-in-one function that generates a full report with dataframes and plots.

Higher Criticism for Word-Frequency Tables

The core statistical engine of this package is the Higher Criticism (HC) test, adapted for textual analysis as described in the work of Kipnis (2022).

In statistical terms, comparing two corpora involves thousands of individual hypothesis tests—one for each unique word. For each word, we can calculate a p-value that represents the probability of observing its frequency difference by pure chance. However, with thousands of tests, we expect many small p-values to occur randomly, a problem known as the multiple testing problem.

Traditional methods like the Bonferroni correction are too conservative in this setting, as they often miss subtle but collectively significant signals. The Higher Criticism statistic, in contrast, is specifically designed to detect a sparse, heterogeneous signal. In the context of stylometry, this translates to the assumption that an author's or model's unique "style" is not defined by a change in every word, but by a small, unknown subset of words that they use idiosyncratically.

The HC statistic works by examining the ordered sequence of all p-values. It seeks the optimal threshold where the number of surprisingly small p-values is most significant relative to what would be expected under the null hypothesis (that there is no difference between the corpora). This makes it exceptionally well-suited for identifying a faint but consistent stylistic "fingerprint" that distinguishes two bodies of text. This package includes the method described in the aforementioned paper to apply this powerful statistical test to word-frequency tables derived from text. The core statistical code is taken from Alon Kipnis's TwoSampleHC GitHub repository.

References

The primary statistical method is based on the work of Donoho and Jin, and its application to text is detailed by Kipnis:

• Kipnis, A. (2022). Higher Criticism for Discriminating Word-Frequency Tables and Authorship Attribution. https://arxiv.org/pdf/1911.01208

• Donoho, D., & Jin, J. (2004). Higher Criticism for detecting sparse heterogeneous mixtures. Annals of Statistics, 32(3), 962-994. https://arxiv.org/pdf/math/0410072

Installation

1. Activate your virtual environment. Make sure you have a Python 3.8+ environment ready and activated.

2. Install from GitHub. You can install the package directly from this GitHub repository.

   pip install "git+https://github.com/oblumner/higher-criticism-stylometry.git#egg=hc_stylometry[display]"
   

Quick Start: How to Use the Code

Here is a simple example of how to use the analyze_and_display function to compare two lists of documents and generate a full report.

from hct_bootstrap import analyze_and_display
import random

#Set random seed for reproducibility
random.seed(42)

background_vocab = [
    "analysis", "system", "data", "model", "research", "method", "result", "time", "user", "network",
    "process", "study", "group", "problem", "level", "information", "effect", "change", "design", "feature",
    "performance", "approach", "case", "point", "development", "variable", "pattern", "value", "algorithm", "test",
    "environment", "structure", "condition", "behavior", "factor", "application", "control", "theory", "practice", "solution",
    "science", "technology", "paper", "review", "framework", "function", "state", "rule", "context", "form",
    "concept", "equation", "experiment", "measure", "source", "type", "area", "base", "number", "size",
    "quality", "property", "task", "technique", "principle", "resource", "material", "energy", "rate", "scale",
    "phase", "stage", "component", "class", "architecture", "mechanism", "relation", "role", "impact", "access"
]

ai_style_words = ["delve", "intricate", "underscore", "meticulous", "crucial", "tapestry", "robust", "realm", "nuance", "leverage"]

human_corpus = []
ai_corpus = []

# 1.  Create 'human' corpus
# Generate 1000 documents, each containing 250 random background words
for _ in range(1000):
    doc = " ".join(random.choices(background_vocab, k=250))
    human_corpus.append(doc)
    
#2.  Create 'AI' corpus 
# Generate 1000 documents of the same length, using the same background vocabulary, but also injecting our 'AI style words' into each document
for _ in range(1000):
    doc = " ".join(random.choices(background_vocab, k=250))
    doc += " " + " ".join(ai_style_words)
    ai_corpus.append(doc)


# 2. Run the analysis and display the report
# We set coupled=False because the lists are not paired one-to-one.  If, for example, you used a list of prompts to generate pairs of text using different LLMs, you would set coupled=True (the corpora would be of equal length)
# We use 1000 bootstrap iterations

results = analyze_and_display(
    corpus1_docs=human_corpus, 
    corpus1_name="Human",
    corpus2_docs=ai_corpus, 
    corpus2_name="AI",
    coupled=False,        
    n_iterations=1000     
)

Example Report Output

Running the code above will generate a report in your console or notebook similar to this:

=== Higher Criticism Analysis Report ===
Comparing: 'Human' vs. 'AI'
Coupled Analysis: False
Bootstrap Iterations: 1000
--------------------------

 --- Corpus Information (Post-Cleaning) ---
# of words in Human: 250,000
# of words in AI: 260,000

--- Higher Criticism Results (Analysis on Full Text) ---

word	p_value	human_frequency (%)	AI_frequency (%)	more_frequent_in
nuance	3.98703e-271	0.0	0.384792	AI
crucial	3.98703e-271	0.0	0.384792	AI
delve	3.98703e-271	0.0	0.384792	AI
intricate	3.98703e-271	0.0	0.384792	AI
leverage	3.98703e-271	0.0	0.384792	AI
tapestry	3.98703e-271	0.0	0.384792	AI
robust	3.98703e-271	0.0	0.384792	AI
meticulous	3.98703e-271	0.0	0.384792	AI
underscore	3.98703e-271	0.0	0.384792	AI
realm	3.98703e-271	0.0	0.384792	AI

(showing top 10 of 17 discriminating words from full data run)

Overall HC Score (Full Data): 4.6882

--- Stable Words Found in 100% of Bootstrap Iterations ---

Words more frequent in 'Human':
None

Words more frequent in 'AI':
meticulous, underscore, nuance, realm, robust, crucial, delve, tapestry, leverage

--- Histogram of Bootstrap Word Count Frequencies ---
(A matplotlib histogram plot is displayed here)

Standard Deviation of Bootstrap HC scores: 0.0095

=== End of Report ===

API Reference

The package exposes two primary functions:

analyze_and_display(...)

The main convenience function for interactive analysis. It performs the full pipeline and prints a formatted report.

• corpus1_docs (List[str]): A list of document strings for the first corpus.
• corpus1_name (str): A descriptive name for the first corpus.
• corpus2_docs (List[str]): A list of document strings for the second corpus.
• corpus2_name (str): A descriptive name for the second corpus.
• coupled (bool): Set to True if corpus1_docs[i] is paired with corpus2_docs[i]. Requires lists to be of the same length.
• n_iterations (int, optional, default: 100): The number of bootstrap iterations to perform for stability analysis.
• random_seed (int, optional, default: 42): Seed for reproducible random sampling.
• pos_tags (List[str], optional): A list of spaCy Part-of-Speech tags to include in the analysis. Defaults to content words like nouns, verbs, adjectives, and adverbs.
• nlp (spacy.Language, optional): A pre-loaded spaCy model. If None, the default_spacy_model will be loaded internally.
• default_spacy_model (str, optional, default: "en_core_web_sm"): The name of the default spaCy model to load if nlp is not provided.

discriminate(...)

The core computational engine. It performs the same analysis as analyze_and_display but returns the raw results as a tuple of (results_dict, cleaned_corpus1_str, cleaned_corpus2_str) without printing or plotting anything. This function is ideal for use in scripts or when you want to handle the output data programmatically. Its parameters are the same as analyze_and_display.