textmeasures

textmeasures is a Python package for quantitative text measurement in corpus linguistics, quantitative linguistics, discourse analysis, syntactic complexity, lexical statistics, collocation analysis, linguistic networks, and embedding-based semantic analysis.

Version 0.2.0 expands the package from frequency-distribution measures into a broader toolkit. Many APIs accept ordinary Python sequences; CoNLL-U based modules use asymgram for dependency treebank parsing.

Researchers are encouraged to use Codex and other AI agents to read the source code, inspect the formulas and references, and then use the relevant metrics directly in their own analysis workflows.

Installation

pip install textmeasures

For local development:

git clone https://github.com/Yihtsy/textmeasures.git
cd textmeasures
pip install -e .

Six Measure Families

textmeasures currently covers six broad families of indicators.

Several functions return bundles of metrics as dictionaries. This design keeps the public API compact while preserving named, stable metric keys.

Quick Start: Frequency Distributions

from textmeasures import FreqDist, entropy, repeat_rate, gini, normalized_entropy

freqs = FreqDist([10, 5, 3, 1, 1])

print(freqs.to_list())
print(entropy(freqs))
print(repeat_rate(freqs))
print(gini(freqs))
print(normalized_entropy(freqs))

Quick Start: Symbol Sequences

from textmeasures import SymbSeq

seq = SymbSeq("the cat saw the dog and the cat")

print(seq.richness())
print(seq.dispersion("the", part_count=3))
print(seq.mtld())

Quick Start: CoNLL-U and Dependency Structure

from textmeasures import dependency_tree_measures, ud_deprel_frequencies

path = "sample.conllu"

tree = dependency_tree_measures(path)
rels = ud_deprel_frequencies(path, denominator="dependencies")

print(tree["mdd"])
print(tree["ndd"])
print(tree["mhd"])
print(rels["absolute"])

CoNLL-U functions usually skip multiword-token and empty-node rows by requiring integer token IDs. Dependency counts and dependency-distance denominators normally exclude root; relation-frequency helpers expose a denominator option so users can divide by dependency count or word count.

Quick Start: Contingency and Association

from textmeasures import TetrachoricTable

table = TetrachoricTable([[10, 2], [3, 20]])

print(table.pmi())
print(table.odds_ratio(correction=0.5))
print(table.fisher_exact())
print(table.summary())

Quick Start: Linguistic Networks

from textmeasures import LingNetwork

net = LingNetwork.from_conllu(
    "sample.conllu",
    network_type="dependency",
    field="lemma",
    directed=True,
    weighted=True,
)

print(net.node_count())
print(net.average_degree())
print(net.metrics())

Quick Start: Embedding-Based Semantic Measures

import numpy as np
from textmeasures import semantic_similarity_features

embeddings = np.random.default_rng(42).normal(size=(8, 384))

features = semantic_similarity_features(embeddings)
print(features["mean_k1"])
print(features["global"])
print(features["graph_density"])

textmeasures does not compute embeddings itself. Pass an n x d matrix from your preferred embedding model, where rows are meaningful units such as words, clauses, sentences, turns, or utterances.

API Overview

Distribution Objects

• FreqDist
• RelFreqDist
• CumFreqDist
• CumRelFreqDist
• FreqSpectrum

Frequency and Vocabulary Measures

Examples include:

• entropy, renyi_entropy, tsallis_entropy, normalized_entropy
• repeat_rate, inverse_repeat_rate, inverse_simpson, simpson
• richness, hill_number, hill_evenness, hill_unevenness
• gini, theil_t, mean_log_deviation, generalized_entropy
• curve_length, lambda_indicator, b1, b2, b3, b4, b5, b6, b8, b10
• h_point, k_point, n_point, m_point, r1, r2, r4
• zipf_fitted_parameters, zipf_mandelbrot_fitted_parameters, zipf_alekseev_fitted_parameters

CoNLL-U Utilities

• conllu_to_symbols
• conllu_to_freqdist
• conllu_token_dict
• conllu_token_is_valid

Supported linguistic units include word, lemma, letter, character, upos, deprel, and n_gram; availability depends on the selected language.

Syntactic and Dependency Measures

• mean_dep_dist
• normalized_dep_dist
• mean_dep_dist_per_sent
• total_dep_dist
• max_dep_dist
• parameterized_mean_dep_dist
• conditional_mean_dep_dist
• dependency_tree_measures
• synstruc_features
• quansyn_features
• linguistic_feature_counts
• linguistic_feature_rates
• ud_deprel_frequencies
• mean_dep_dist_by_deprel

Sequence Measures

• SymbSeq.richness()
• SymbSeq.dispersion()
• SymbSeq.dispersion_table()
• PosSeq.pos_metrics()
• NumSeq.summary()
• Motif.motif_metrics()

Network and Semantic Measures

• LingNetwork.from_edges()
• LingNetwork.from_conllu()
• LingNetwork.metrics()
• LingNetwork.micro_metrics()
• LingNetwork.macro_metrics()
• LingNetwork.distribution_metrics()
• semantic_similarity_features
• semantic_graph_metrics
• temporal_dynamics

Inputs and Return Values

• Frequency-distribution functions accept integer counts, probability vectors, relative-frequency vectors, or FreqDist-like objects.
• CoNLL-U functions accept paths ending in .conllu, asymgram.TokenList, or asymgram.SentenceList objects, depending on the function.
• Network functions use NetworkX graphs internally but expose a linguistic wrapper class.
• Embedding functions accept two-dimensional numeric arrays.
• Bundle functions return dictionaries with stable snake_case keys.

Citation

If you use textmeasures in academic work, please cite it as research software.

APA

Yih, T. (2026). textmeasures: Quantitative text measurement tools for frequency, sequence, syntax, contingency, network, and semantic measures (Version 0.2.0) [Computer software]. GitHub. https://github.com/Yihtsy/textmeasures

BibTeX

@software{yih_textmeasures_2026,
  author = {Yih, Tsy},
  title = {textmeasures: Quantitative text measurement tools for frequency, sequence, syntax, contingency, network, and semantic measures},
  year = {2026},
  version = {0.2.0},
  url = {https://github.com/Yihtsy/textmeasures},
  note = {Python package}
}

Development Notes

This package is still evolving. New metrics are added conservatively: formulas should be explicit, input requirements should be clear, and public functions should include references in their docstrings.

License

License information will be added here.