ppmi-calc 0.1.1

A Python library for calculating Pointwise Mutual Information (PMI)

PPMI - Pointwise Mutual Information Library

A Python library for calculating Pointwise Mutual Information (PMI), Positive PMI (PPMI), and Normalized PMI (NPMI) between events or words.

Overview

Pointwise Mutual Information (PMI) is a measure of association used in information theory and statistics. It measures how much more two events (such as words) co-occur than we would expect them to by chance if they were independent.

Intuition: PMI asks "how much more do these two words appear together in our data than we would expect if they were unrelated?" A high PMI means the words appear together much more often than random chance would predict, indicating a strong association.

Common Use: PMI is particularly useful for building term-term matrices (word-word co-occurrence matrices), where vector dimensions correspond to words rather than documents. This makes it valuable for discovering semantic relationships and word associations.

Formula

PMI(x, y) = log(P(x,y) / (P(x) × P(y)))

Where:

• P(x,y) is the joint probability of x and y
• P(x) is the marginal probability of x
• P(y) is the marginal probability of y

Features

• PMI Calculation: Standard pointwise mutual information
• PPMI: Positive PMI (max(0, PMI)) to handle negative associations
• Weighted PPMI: Context distribution smoothing with alpha parameter (recommended: α=0.75)
• NPMI: Normalized PMI ranging from -1 to 1
• Flexible Input: Calculate from probabilities, counts, or observations
• Batch Processing: Add multiple observations at once
• Matrix Export: Convert results to numpy matrices
• Multiple Logarithm Bases: Support for base 2, 10, e, or any custom base

Installation

From PyPI

pip install ppmi-calc

From source

git clone https://github.com/yourusername/ppmi.git
cd ppmi
pip install -e .

For development

pip install -e ".[dev]"

Quick Start

Using the PMI Class

from ppmi import PMI

# Create a PMI calculator
pmi_calc = PMI()

# Add observations
pmi_calc.add_observation("word1", "word2")
pmi_calc.add_observation("word1", "word3")
pmi_calc.add_observation("word2", "word3", count=5)

# Calculate PMI for a specific pair
pmi_value = pmi_calc.calculate_pmi("word1", "word2")
print(f"PMI(word1, word2) = {pmi_value}")

# Calculate PPMI
ppmi_value = pmi_calc.calculate_ppmi("word1", "word2")
print(f"PPMI(word1, word2) = {ppmi_value}")

# Get all PMI scores
all_pmi = pmi_calc.get_all_pmi()
for (x, y), score in all_pmi.items():
    print(f"PMI({x}, {y}) = {score}")

Using Standalone Functions

from ppmi import calculate_pmi_from_probabilities, calculate_ppmi_from_probabilities

# Calculate from probabilities
pmi = calculate_pmi_from_probabilities(p_xy=0.1, p_x=0.3, p_y=0.4, base=2)
ppmi = calculate_ppmi_from_probabilities(p_xy=0.1, p_x=0.3, p_y=0.4, base=2)

print(f"PMI = {pmi}")
print(f"PPMI = {ppmi}")

Calculate from Counts

from ppmi.pmi import calculate_pmi_from_counts

# Raw counts
co_occurrence_count = 10  # times x and y appeared together
x_count = 50              # total occurrences of x
y_count = 40              # total occurrences of y
total_observations = 1000

pmi = calculate_pmi_from_counts(
    co_count=co_occurrence_count,
    x_count=x_count,
    y_count=y_count,
    total=total_observations
)
print(f"PMI = {pmi}")

Batch Processing

from ppmi import PMI

pmi_calc = PMI()

# Add multiple observations at once
pairs = [
    ("apple", "fruit"),
    ("banana", "fruit"),
    ("carrot", "vegetable"),
    ("apple", "red", 3),  # with count
]

pmi_calc.add_observations_batch(pairs)

# Get all PPMI scores
all_ppmi = pmi_calc.get_all_ppmi()

Export to Matrix

import numpy as np
from ppmi import PMI

pmi_calc = PMI()
# ... add observations ...

# Get PPMI as a matrix
matrix, x_labels, y_labels = pmi_calc.to_matrix(metric='ppmi', base=2)

print("PPMI Matrix:")
print(matrix)
print(f"X items: {x_labels}")
print(f"Y items: {y_labels}")

Weighted PPMI with Context Distribution Smoothing

The library supports weighted PPMI with the alpha parameter for context distribution smoothing. This is particularly useful for word embeddings and reduces bias toward rare contexts.

Formula:

PPMIα(w,c) = max(0, log(P(w,c) / (P(w) × Pα(c))))
where Pα(c) = count(c)α / Σ count(c)α

Usage:

from ppmi import PMI

pmi_calc = PMI()
# ... add observations ...

# Standard PPMI (alpha=1.0, default)
ppmi_standard = pmi_calc.calculate_ppmi("word", "context", alpha=1.0)

# Weighted PPMI with recommended alpha=0.75
ppmi_weighted = pmi_calc.calculate_ppmi("word", "context", alpha=0.75)

# Get all PPMI scores with smoothing
all_ppmi = pmi_calc.get_all_ppmi(alpha=0.75)

# Export to matrix with smoothing
matrix, words, contexts = pmi_calc.to_matrix(metric='ppmi', alpha=0.75)

Effect of alpha:

• alpha = 1.0: Standard PPMI (no smoothing)
• alpha < 1.0 (e.g., 0.75): Increases probability of rare contexts, reducing their PPMI
• alpha > 1.0: Decreases probability of rare contexts, increasing their PPMI

Recommendation: Levy et al. (2015) found that alpha=0.75 improves performance of embeddings on a wide range of tasks. This works by reducing the bias toward rare co-occurrences.

Use Cases

Natural Language Processing

• Word co-occurrence analysis
• Collocation detection
• Feature extraction for word embeddings
• Measuring word associations

Information Retrieval

• Query expansion
• Document similarity
• Term weighting

Data Mining

• Association rule mining
• Feature selection
• Pattern discovery

API Reference

PMI Class

Methods

• __init__(): Initialize the PMI calculator
• add_observation(x, y, count=1): Add a single observation
• add_observations_batch(pairs): Add multiple observations
• calculate_pmi(x, y, base=2): Calculate PMI for a pair
• calculate_ppmi(x, y, base=2, alpha=1.0): Calculate Positive PMI with optional context smoothing
• calculate_npmi(x, y, base=2): Calculate Normalized PMI
• get_all_pmi(base=2): Get PMI for all pairs
• get_all_ppmi(base=2, alpha=1.0): Get PPMI for all pairs with optional smoothing
• to_matrix(metric='ppmi', base=2, alpha=1.0): Export to numpy matrix

Standalone Functions

• calculate_pmi_from_probabilities(p_xy, p_x, p_y, base=2): Calculate PMI from probabilities
• calculate_ppmi_from_probabilities(p_xy, p_x, p_y, base=2): Calculate PPMI from probabilities
• calculate_pmi_from_counts(co_count, x_count, y_count, total, base=2): Calculate PMI from counts
• calculate_ppmi_from_counts(co_count, x_count, y_count, total, base=2): Calculate PPMI from counts

Parameters

• base: Logarithm base. Can be:

  • 2: Binary logarithm (default)
  • 10: Common logarithm
  • 'e' or math.e: Natural logarithm
  • Any positive number: Custom base

• alpha: Context distribution smoothing parameter (for PPMI). Can be:

  • 1.0: No smoothing, standard PPMI (default)
  • < 1.0 (e.g., 0.75): Recommended for word embeddings. Increases probability of rare contexts.
  • > 1.0: Decreases probability of rare contexts.

  Reference: Levy et al. (2015) "Improving Distributional Similarity with Lessons Learned from Word Embeddings"

Examples

See the examples/ directory for more detailed examples:

• basic_usage.py: Basic PMI calculations
• text_analysis.py: Word co-occurrence analysis
• weighted_ppmi_example.py: Weighted PPMI with context smoothing

Testing

Run tests with pytest:

pytest tests/

With coverage:

pytest --cov=ppmi tests/

Requirements

• Python >= 3.7
• numpy >= 1.19.0

License

MIT License - see LICENSE file for details

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Citation

If you use this library in your research, please cite:

@software{ppmi,
  author = {Kossymova, Gulshat},
  title = {PPMI: A Python Library for Pointwise Mutual Information},
  year = {2026},
  url = {https://github.com/guls-1/ppmi}
}

References

• Church, K. W., & Hanks, P. (1990). Word association norms, mutual information, and lexicography. Computational linguistics, 16(1), 22-29.
• Bouma, G. (2009). Normalized (pointwise) mutual information in collocation extraction. Proceedings of GSCL, 31-40.