google-ngrams 0.2.0

Fetch and analyze Google Ngram data for specified word forms.

This package has functions for processing Google’s Ngram repositories without having to download them locally. These repositories vary in their size, but the larger ones (like th one for the letter s or common bigrams) contain multiple gigabytes.

The main function uses scan_csv from the polars package to reduce memory load. Still, depending on the specific word forms being searched, loading and processing the data tables can sometimes take a few minutes if they are large.

vnc

To analyze the returned data, the package also contains functions based on the work of Gries and Hilpert (2012) for Variability-Based Neighbor Clustering.

The idea is to use hierarchical clustering to aid “bottom up” periodization of language change. The python functions are built on their original R code.

Distances, therefore, are calculated in sums of standard deviations and coefficients of variation, according to their stated method.

Dendrograms are plotted using matplotlib, with custom implementations for hierarchical clustering that maintain the plotting order of the leaves according to the requirements of the method.

The package also has a custom implementation of dendrogram truncation that consolidates leaves under a specified number of time periods (or clusters) while also maintaining the leaf order to facilitate the reading and interpretation of large dendrograms.

Lightweight Implementation

Starting with version 0.2.0, google_ngrams uses lightweight, custom implementations for statistical computations instead of heavy dependencies like scipy and statsmodels. This design choice reduces installation overhead while maintaining full functionality for the core VNC methodology and smoothing operations.

Installation

You can install the released version of google_ngrams from PyPI:

pip install google-ngrams

Usage

To use the google_ngrams package, import google_ngram to fetch data and TimeSeries for analysis.

from google_ngrams import google_ngram, TimeSeries

Fetching n-gram data

The google_ngram function supports different varieties of English (e.g., British, American) and allows aggregation by year or decade. Word forms (even a single word form) must be formatted as a list:

The following would return counts for the word x-ray in US English by year:

xray_year = google_ngram(word_forms = ["x-ray"], variety = "us", by = "year")

Alternatively, the following would return counts of the combined forms xray and xrays in British English by decade:

xray_decade = google_ngram(word_forms = ["x-ray", "x-rays"], variety = "gb", by = "decade")

The function returns a polars DataFrame with either a time interval column (either Year or Decade) and columns for Token, AF (absolute frequency) and RF (relative frequency).

The returned DataFrame, then, can be manipulated using the polars API:

import polars as pl

xray_filtered = xray_decade.filter(pl.col("Decade") >= 1900)

Analyzing time series data

To analyze the data, use TimeSeries, specifying a column of time intervals and a column of relative frequencies:

xray_ts = TimeSeries(xray_filtered, time_col="Decade", values_col="RF")

VNC dendrograms can then be plotted with a variety of options:

xray_ts.timeviz_vnc()

For additional information, consult the documentation.

License

Code licensed under MIT License. See LICENSE file.