Skip to main content

A natural language parser for Icelandic

Project description

https://github.com/mideind/GreynirPackage/blob/master/doc/_static/GreynirLogo220.png?raw=true

A fast, efficient natural language processor for Icelandic

https://github.com/mideind/GreynirPackage/workflows/Python%20package/badge.svg?branch=master

Overview

Greynir is a Python 3 (>= 3.6) package for working with Icelandic natural language text. Greynir can parse text into sentence trees, find lemmas, inflect noun phrases, assign part-of-speech tags and much more.

Greynir’s sentence trees can inter alia be used to extract information from text, for instance about people, titles, entities, facts, actions and opinions.

Full documentation for Greynir is available here.

Greynir is the engine of Greynir.is, a natural-language front end for a database of over 10 million sentences parsed from Icelandic news articles, and Embla, a voice-driven virtual assistant app for smart devices such as iOS and Android phones.

Greynir includes a hand-written context-free grammar for the Icelandic language, consisting of over 7,000 lines of grammatical productions in extended Backus-Naur format. Its fast C++ parser core is able to cope with long and ambiguous sentences, using an Earley-type parser as enhanced by Scott and Johnstone.

Greynir employs the Tokenizer package, by the same authors, to tokenize text.

Examples

Use Greynir to easily inflect noun phrases:

from reynir import NounPhrase as Nl

# Create a NounPhrase ('nafnliður') object
karfa = Nl("þrír lúxus-miðar á Star Wars og tveir brimsaltir pokar af poppi")

# Print the NounPhrase in the correct case for each context
# (þf=þolfall/accusative, þgf=þágufall/dative). Note that
# the NounPhrase class implements __format__(), allowing you
# to use the case as a format specification, for instance in f-strings.

print(f"Þú keyptir {karfa:þf}.")
print(f"Hér er kvittunin þín fyrir {karfa:þgf}.")

The program outputs the following text, correctly inflected:

Þú keyptir þrjá lúxus-miða á Star Wars og tvo brimsalta poka af poppi.
Hér er kvittunin þín fyrir þremur lúxus-miðum á Star Wars og tveimur brimsöltum pokum af poppi.

Use Greynir to parse a sentence:

>>> from reynir import Greynir
>>> g = Greynir()
>>> sent = g.parse_single("Ása sá sól.")
>>> print(sent.tree.view)
P                               # Root
+-S-MAIN                        # Main sentence
  +-IP                          # Inflected phrase
    +-NP-SUBJ                   # Noun phrase, subject
      +-no_et_nf_kvk: 'Ása'     # Noun, singular, nominative, feminine
    +-VP                        # Verb phrase containing arguments
      +-VP                      # Verb phrase containing verb
        +-so_1_þf_et_p3: 'sá'   # Verb, 1 accusative arg, singular, 3rd p
      +-NP-OBJ                  # Noun phrase, object
        +-no_et_þf_kvk: 'sól'   # Noun, singular, accusative, feminine
+-'.'                           # Punctuation
>>> sent.tree.nouns
['Ása', 'sól']
>>> sent.tree.verbs
['sjá']
>>> sent.tree.flat
'P S-MAIN IP NP-SUBJ no_et_nf_kvk /NP-SUBJ VP so_1_þf_et_p3
    NP-OBJ no_et_þf_kvk /NP-OBJ /VP /IP /S-MAIN p /P'
>>> # The subject noun phrase (S.IP.NP also works)
>>> sent.tree.S.IP.NP_SUBJ.lemmas
['Ása']
>>> # The verb phrase
>>> sent.tree.S.IP.VP.lemmas
['sjá', 'sól']
>>> # The object within the verb phrase (S.IP.VP.NP also works)
>>> sent.tree.S.IP.VP.NP_OBJ.lemmas
['sól']

Prerequisites

This package runs on CPython 3.6 or newer, and on PyPy 3.6 or newer.

To find out which version of Python you have, enter:

$ python --version

If a binary wheel package isn’t available on PyPi for your system, you may need to have the python3-dev package (or its Windows equivalent) installed on your system to set up Greynir successfully. This is because a source distribution install requires a C++ compiler and linker:

$ # Debian or Ubuntu:
$ sudo apt-get install python3-dev

Depending on your system, you may also need to install libffi-dev:

$ # Debian or Ubuntu
$ sudo apt-get install libffi-dev

Installation

To install this package, assuming Python 3 is your default Python:

$ pip install reynir

If you have git and git-lfs installed and want to be able to edit the source, do like so:

$ git clone https://github.com/mideind/GreynirPackage
$ cd GreynirPackage
$ # [ Activate your virtualenv here if you have one ]
$ git lfs install
$ git pull
$ pip install -e .

The package source code is now in GreynirPackage/src/reynir.

Note that git-lfs is required to clone and pull the full compressed binary files for the Beygingarlýsing íslensks nútímamáls (BÍN) database. If it is missing, you will get assertion errors when you try to run Greynir.

Tests

To run the built-in tests, install pytest, cd to your GreynirPackage subdirectory (and optionally activate your virtualenv), then run:

$ python -m pytest

Evaluation

A parsing test pipeline for different parsing schemas, including the Greynir schema, has been developed. It is available here.

Documentation

Please consult Greynir’s documentation for detailed installation instructions, a quickstart guide, and reference information, as well as important information about copyright and licensing.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for reynir, version 2.10.1
Filename, size File type Python version Upload date Hashes
Filename, size reynir-2.10.1-cp36-cp36m-manylinux1_x86_64.whl (39.2 MB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size reynir-2.10.1-cp36-cp36m-manylinux2010_x86_64.whl (39.2 MB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size reynir-2.10.1-cp37-cp37m-manylinux1_x86_64.whl (39.2 MB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size reynir-2.10.1-cp37-cp37m-manylinux2010_x86_64.whl (39.2 MB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size reynir-2.10.1-cp38-cp38-manylinux1_x86_64.whl (39.2 MB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size reynir-2.10.1-cp38-cp38-manylinux2010_x86_64.whl (39.2 MB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size reynir-2.10.1-cp39-cp39-manylinux1_x86_64.whl (39.2 MB) File type Wheel Python version cp39 Upload date Hashes View
Filename, size reynir-2.10.1-cp39-cp39-manylinux2010_x86_64.whl (39.2 MB) File type Wheel Python version cp39 Upload date Hashes View
Filename, size reynir-2.10.1-pp36-pypy36_pp73-manylinux1_x86_64.whl (39.1 MB) File type Wheel Python version pp36 Upload date Hashes View
Filename, size reynir-2.10.1-pp36-pypy36_pp73-manylinux2010_x86_64.whl (39.1 MB) File type Wheel Python version pp36 Upload date Hashes View
Filename, size reynir-2.10.1-pp36-pypy3_72-manylinux1_x86_64.whl (39.1 MB) File type Wheel Python version pp36 Upload date Hashes View
Filename, size reynir-2.10.1-pp36-pypy3_72-manylinux2010_x86_64.whl (39.1 MB) File type Wheel Python version pp36 Upload date Hashes View
Filename, size reynir-2.10.1-pp37-pypy37_pp73-manylinux1_x86_64.whl (39.1 MB) File type Wheel Python version pp37 Upload date Hashes View
Filename, size reynir-2.10.1-pp37-pypy37_pp73-manylinux2010_x86_64.whl (39.1 MB) File type Wheel Python version pp37 Upload date Hashes View
Filename, size reynir-2.10.1.tar.gz (39.0 MB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page