TermXtract

TermXtract is a Python library for automatic term extraction (ATE): finding the domain-specific terms in a collection of documents. It offers a range of classic unsupervised scoring methods (TF-IDF, C-Value, Weirdness, RAKE, topic models, …) and supervised methods that learn from labelled examples (AdaBoost, Roger, and transformer-based token classification).

Every method takes either a plain list of strings or a Teanga Corpus, and returns the same ATEResults object.

📖 Documentation: https://teanga.io/TermXtract/
📝 Changelog: CHANGELOG.md. TermXtract follows Semantic Versioning; before 1.0, minor releases may contain breaking changes.

Installation

TermXtract needs Python 3.10 or later. It is not yet on PyPI, so install it from GitHub:

pip install git+https://github.com/TeangaNLP/TermXtract.git

The transformer-based token-classification extractor needs PyTorch and Hugging Face Transformers, which are large, so they are an optional extra. Install them with:

pip install "termxtract[transformers] @ git+https://github.com/TeangaNLP/TermXtract.git"

All other methods work with the base install.

Quickstart

Unsupervised extraction

from termxtract import TermExtractor

corpus = [
    "Machine learning is a field of artificial intelligence.",
    "Deep learning is a subfield of machine learning.",
    "Natural language processing applies machine learning to text.",
]

extractor = TermExtractor(method="cvalue", n=2)  # consider unigrams and bigrams
results = extractor.extract(corpus)

for doc in results.terms:
    print(doc["doc_id"], doc["terms"][:3])
# doc_0 [{'term': 'machine learning', 'score': 1.0}, {'term': 'artificial intelligence', 'score': 1.0}, {'term': 'machine', 'score': 0.0}]
# ...

Each document's terms are ranked from highest to lowest score.

Pass threshold=... to keep only terms scoring at or above a minimum.

Candidates that start or end with a stopword, such as is a or of machine, are dropped by default, using a built-in list of English function words. Stopwords inside a term, as in degree of freedom, are kept. Pass stopwords=None to keep every candidate, or a list of words to use your own stopwords.

If a Teanga corpus has part-of-speech tags, pos_pattern="noun_phrase" keeps only noun phrases, such as machine learning or degree of freedom (see Using a Teanga corpus).

Methods that need a reference corpus

weirdness, domainpertinence and relevance compare the target corpus against a general-language reference corpus:

reference = [
    "The weather was pleasant and people walked in the park.",
    "She bought bread and milk at the shop on the corner.",
]

extractor = TermExtractor(method="weirdness", reference_corpus=reference, n=2)
results = extractor.extract(corpus)

Supervised extraction

Supervised methods are trained from a dictionary of candidate terms labelled 1 (term) or 0 (not a term):

from termxtract import SupervisedTermExtractor

labels = {
    "machine learning": 1,
    "deep learning": 1,
    "natural language processing": 1,
    "field": 0,
    "text": 0,
}

extractor = SupervisedTermExtractor(method="adaboost", n=3)
results = extractor.extract(corpus, labels)

The AdaBoost extractor uses Open English WordNet through the wn package. The first time it needs WordNet it downloads the oewn:2025+ lexicon into ~/.wn_data (set WN_DATA_DIR to change this). To install it ahead of time, e.g. for offline use, run python -m wn download oewn:2025+. Pass wordnet_lexicon=... to use a different lexicon.

Using a Teanga corpus

Any extractor also accepts a Teanga Corpus with a words span layer:

from teanga import Corpus

corpus = Corpus()
corpus.add_layer_meta("text")
corpus.add_layer_meta("words", layer_type="span", base="text")
doc = corpus.add_doc("Deep learning is a subfield of machine learning.")
doc.words = [(0, 4), (5, 13), (14, 16), (17, 18), (19, 27), (28, 30), (31, 38), (39, 47)]

results = TermExtractor(method="basic", n=2).extract(corpus)

Words are read from the words layer or, if there is none, from a tokens layer, as written by Teanga's CoNLL-U reader and spaCy service; pass words_layer=... to use another layer. If the corpus has part-of-speech tags in a upos or pos layer based on the words, pass pos_pattern to keep only the terms whose tags match a pattern, either "noun_phrase" or your own pattern such as "<ADJ|NOUN>*<NOUN>". See the TF-IDF tutorial.

Available methods

TermExtractor.METHODS and SupervisedTermExtractor.METHODS map each method name to its implementation, so list(TermExtractor.METHODS) lists the available methods. Passing an unknown method raises a ValueError that lists them too.

Unsupervised — TermExtractor(method=...)

All methods accept n (maximum n-gram length), threshold (minimum score), stopwords (default "english"), and, for Teanga input, words_layer, pos_pattern and pos_layer. rake does not use n, stopwords or words_layer; it splits candidates at its own stoplist.

Supervised — SupervisedTermExtractor(method=...)

All supervised methods accept n (maximum n-gram length, default 6), and stopwords and pos_pattern, which filter the extracted terms but not the training candidates, and words_layer and pos_layer for Teanga input.

Results

extract() returns an ATEResults object with:

• results.terms: one entry per document, {"doc_id": ..., "terms": [{"term": str, "score": float}, ...]}, with terms ranked from highest to lowest score. Every method uses this format.
• results.offsets: for Teanga input, one entry per document, {"doc_id": ..., "offsets": [{"term": str, "offset": (start, end)}, ...]}, giving the character offsets of every occurrence of each extracted term. It is None for plain-text input, and for nmf and domaincoherence, which score terms over the whole corpus rather than per document.
• results.corpus: the input corpus.

Development

git clone https://github.com/TeangaNLP/TermXtract.git
cd TermXtract
poetry install --with dev --extras transformers

poetry run pytest                              # run the test suite
poetry run pytest --cov                        # ...with a coverage report
poetry run ruff check termxtract tests         # lint
poetry run ruff format termxtract tests        # format
poetry run mypy termxtract                     # type check
poetry run mkdocs serve                        # preview the docs locally

CI runs all of these on every pull request, with the tests on Python 3.10–3.12, on Linux, macOS and Windows, and against the oldest dependency versions pyproject.toml allows. The codebase was reformatted in one commit, which git blame can skip with git config blame.ignoreRevsFile .git-blame-ignore-revs.

Releasing

Version numbers follow Semantic Versioning: bump the patch version for bug fixes, the minor version for new features, and the major version for breaking changes (before 1.0, breaking changes bump the minor version).

1. In CHANGELOG.md, rename ## [Unreleased] to ## [X.Y.Z] - YYYY-MM-DD, add a new empty ## [Unreleased] section above it, and update the comparison links at the bottom of the file.
2. Set the version with poetry version X.Y.Z (or poetry version patch, minor or major).
3. Commit both files as Release X.Y.Z and get the commit onto main.
4. On GitHub, draft a release with a new tag vX.Y.Z on that commit, paste the changelog section as the release notes, and publish it. The release workflow checks that the tag matches the version, builds the package and uploads it to PyPI.

pyproject.toml is still at 0.1.0 and nothing has been published, so the first release can be 0.1.0 without a version bump.

License

TermXtract is licensed under the Apache License 2.0, the same license as Teanga.