patternminer: Mining Recurrent Text Patterns in a Corpus

patternminer makes it easy to mine recurrent surface text patterns in a corpus of texts: which token subsequences recur, where, how often, and how they nest inside each other.

It is built on pygstlib (generalized suffix trees — mining is linear-time) and generalizes the pattern-mining approach of dialign / pydialign beyond dyadic dialogue.

Features:

• cross-document mining: patterns present in ≥ k documents, with every occurrence located as (doc_id, unit_id, start) and both absolute (doc_freq, unit_freq) and relative (doc_support, unit_support) frequencies;
• within-document mining: patterns a document repeats across its own units (sentences, lines, utterances…);
• hierarchy: regroup subpatterns under their parent patterns (e.g. parent hello world ! groups hello and world !);
• filtering: by size, frequency, arbitrary predicates, or down to the maximal patterns;
• text in, patterns out: built-in basic tokenizers and unit splitters — or bring your own tokens;
• pandas layer (optional): patterns and occurrences as DataFrames.

Installation

uv add patternminer               # or: pip install patternminer
uv add 'patternminer[analysis]'   # with the pandas layer

Requires Python ≥ 3.10. The only required runtime dependency is pygstlib.

Quickstart

A corpus is a list of documents; each document is a list of units (the granularity at which repetition is observed — by default, one unit per line); each unit is a tuple of string tokens.

from patternminer import Corpus, mine_corpus

corpus = Corpus.from_texts([
    "hello world !\nhow are you today ?",
    "hello world ! how are you doing ?",
])
patterns = mine_corpus(corpus)  # patterns shared by >= 2 documents

for p in patterns:
    print(f"{p.surface!r}: doc_freq={p.doc_freq}, occurrences={list(p.occurrences)}")

print()
print(patterns.hierarchy().render())

Output:

'hello world !': doc_freq=2, occurrences=[(0, 0, 0), (1, 0, 0)]
'how are you': doc_freq=2, occurrences=[(0, 1, 0), (1, 0, 3)]
'?': doc_freq=2, occurrences=[(0, 1, 4), (1, 0, 7)]

hello world !  [doc_freq=2, unit_freq=2]
how are you  [doc_freq=2, unit_freq=2]
?  [doc_freq=2, unit_freq=2]

Subpatterns that never occur independently (are you, world !, you, ! — always enclosed in hello world ! / how are you) are discarded; they would appear if they occurred free somewhere in the corpus.

Tour of the API

from patternminer import Corpus, mine_corpus, mine_document, mine_within, nlp

# Corpus building — raw text with basic helpers…
corpus = Corpus.from_texts(texts, tokenizer=nlp.word_punct_tokenizer,
                           unit_splitter=nlp.split_sentences, lowercase=True)
corpus = Corpus.from_files(paths)
# …or pre-tokenized input:
corpus = Corpus.from_token_lists(list_of_token_lists)   # 1 unit per document
corpus = Corpus.from_token_units(docs_units_tokens)     # nested

# Mining — threshold by absolute freq or relative support (or both)
patterns = mine_corpus(corpus, min_doc_freq=2, is_valid=nlp.has_alphabetic)
patterns = mine_corpus(corpus, min_doc_support=0.5)   # >= half the documents
repetitions = mine_document(corpus[0])       # within one document
per_doc = mine_within(corpus, min_unit_support=0.1)   # doc_id -> PatternSet

# Pattern: frequencies, supports, free/constrained classification
p = patterns.get("world !")
p.doc_freq; p.unit_freq    # absolute counts (documents / units)
p.doc_support              # doc_freq / total documents in the corpus
p.unit_support             # unit_freq / total units in the corpus
p.occurrences              # every occurrence, (doc_id, unit_id, start)
p.free_occurrences         # the subset not enclosed in a larger pattern
p.constrained_occurrences  # the enclosed complement

# PatternSet
patterns.filter(min_size=2, min_unit_freq=3, min_doc_support=0.5, ...)
patterns.maximal()          # patterns contained in no other
patterns.get("world !")     # lookup by surface or token tuple; None if the
                            # key is absent (e.g. a pattern that never
                            # occurs free — see caveats below)
patterns.to_dataframe()     # pandas ([analysis] extra)

# Hierarchy (containment DAG)
hier = patterns.hierarchy()
hier.roots; hier.children(p); hier.parents(p); hier.descendants(p)
print(hier.render())

➡ Tutorials: the quickstart notebook and a real corpus study on three chapters of David Copperfield.

Semantics & caveats

• Free patterns only. The inventory holds the right-maximal repeats that occur free at least once; a pattern whose every occurrence is enclosed in a free occurrence of a larger pattern is discarded (e.g. are you is dropped when it only ever appears inside how are you). Kept patterns still report all their occurrences, enclosed instances included; Pattern.free_occurrences / constrained_occurrences expose the per-occurrence split. Details in docs/architecture.md.
• Repetition is counted across units. A pattern occurring twice inside a single unit (and nowhere else) is not detected. Choose the unit granularity accordingly (nlp.split_sentences, nlp.split_lines, …).
• Surface-level. patternminer sees exactly the tokens you give it — normalize (case, tokenization) upstream, or use the lowercase=True and tokenizer options. The built-in tokenizers are deliberately simple.
• Determinism. Results come in a canonical order (size desc, then tokens) and are identical across processes.
• Concurrency. Like pygstlib, mining structures are not thread-safe; the returned Pattern/PatternSet/PatternHierarchy objects are immutable and safe to share once built.

Development

uv venv && uv pip install -e '.[dev]' --group notebook
uv run pytest          # unit, property, randomized naive-reference and notebook tests
uv run ruff check .

The project is managed BMAD-style: see docs/prd.md, docs/architecture.md and the story backlog in docs/stories/.

Contributors

• Guillaume Dubuisson Duplessis (2026)

Usage for Research Purposes

If you use this library for research purposes, please make reference to it by citing the following paper on recurrent surface text patterns:

• Dubuisson Duplessis, G.; Charras, F.; Letard, V.; Ligozat, A.-L.; Rosset, S., Utterance Retrieval based on Recurrent Surface Text Patterns, 39th European Conference on Information Retrieval (ECIR), 2017, pp. 199--211 [More DOI]

See also dialign and pydialign for the dialogue-specific measures built on the same mining approach.

License

MIT — see the LICENSE.txt file.