Skip to main content

A lemmatizer for German language text.

Project description

GermaLemma

December 2019, Markus Konrad markus.konrad@wzb.eu / Berlin Social Science Center

A lemmatizer for German language text

Germalemma lemmatizes Part-of-Speech-tagged German language words. To do so, it combines a large lemma dictionary (an excerpt of the TIGER corpus from the University of Stuttgart), functions from the CLiPS "Pattern" package, and an algorithm to split composita.

Installation

Easy option: Installing from PyPI via pip

You can install the package from PyPI via pip:

pip install -U germalemma

Downloading and installing from source

In order to use GermaLemma, you will need to install some additional packages (see Requirements section below) and then download the TIGER corpus from the University of Stuttgart. You will need to use the CONLL09 format, not the XML format. The corpus is free to use for non-commercial purposes (see License Agreement).

Then, you should convert the corpus into pickle format for faster loading by executing germalemma.py and passing the path to the corpus file in CONLL09 format:

python germalemma.py tiger_release_[...].conll09

This will place a lemmata.pickle file in the data directory which is then automatically loaded.

Part-of-Speech (POS) Tagging

You will need to apply Part-of-Speech (POS) tagging to your text before you can lemmatize its words. See this blog post on how to do that.

Usage

You have set up GermaLemma to use the TIGER corpus (as explained above). You have tokenized your text (e.g. with NLTK). You have POS-tagged your tokens. Now you can use GermaLemma:

from germalemma import GermaLemma

lemmatizer = GermaLemma()

# passing the word and the POS tag ("N" for noun)
lemma = lemmatizer.find_lemma('Feinstaubbelastungen', 'N')
print(lemma)
# -> lemma is "Feinstaubbelastung"

Valid POS tags

You can pass POS tags from the STTS tagset, however, only four POS tags can be processed:

  • 'N...' (nouns)
  • 'V...' (verbs)
  • 'ADJ...' (adjectives)
  • 'ADV...' (adverbs)

All other POS tags will result in a ValueError so you should wrap the call to find_lemma in a try-except block.

Accuracy

GermaLemma's accuracy was evaluated using a sample of 696 POS tagged and manually lemmatized words from a sample of paragraphs from proceedings of the European Parliament, Goethe's "Werther", Kafka's "Verwandlung" and a news article from the website of the WZB (see samples in folder "eval_texts").

Under the assumption that the POS tag is correct (only those words were selected), GermaLemma finds the correct lemma in 99.43% of the cases. For comparison, Pattern achieved 95.11% for the same sample.

Requirements

  • Python 3.6 or newer
  • required package Pyphen
  • optional package PatternLite (This package is optional but highly recommended as it boosts the lemmatizer's accuracy.)

License

Apache License 2.0. See LICENSE file.

The TIGER corpus is not part of this repository and has to be downloaded separately under separate license conditions.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for germalemma, version 0.1.3
Filename, size File type Python version Upload date Hashes
Filename, size germalemma-0.1.3-py3-none-any.whl (2.3 MB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size germalemma-0.1.3.tar.gz (2.3 MB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page