inferential-analysis 2.0.0

Pure-Python inferential analysis

Inferential Analysis

The book "Validity, Reliability, and Significance: Empirical Methods for NLP and Data Science", written by Stefan Riezler and Michael Hagmann, uses the inferential analysis module found at https://github.com/StatNLP/empirical_methods to conduct inferential reproducibility analyses.

The aim of this project is to offer functionality similar to that of the original inferential_analysis module, but without depending on R and lme4.

Installation

The library can be found on PyPI.

# pip
pip install inferential-analysis

# uv
uv add inferential-analysis

Usage Examples

The usage examples are based on this Jupyter notebook and can be found in the examples directory of the repository.

Limitations

Reliability

The current library does not allow analyzing per-example variance components (see issue #4). In case that is required, you can fall back to R and lme4 like so:

library(lme4)
data <- read.csv("evaluation_data/aghajanyan_cnn-all.csv")
model <- lmer(
  rouge_2 ~ 1 + (1|seed) + (1|distribution) + (1|lambda) + (1|summary_id),
  data = data[data$system == "SOTA",]
)
summary(model)

Additional Information

• Changes are tracked in the Changelog

License

In accordance with the original implementation this library is licensed under the Apache 2.0 license.