tokenizer 1.0.1

A tokenizer for Icelandic text

Overview

Tokenization is a necessary first step in many natural language processing tasks, such as word counting, parsing, spell checking, corpus generation, and statistical analysis of text.

Tokenizer is a compact pure-Python (2 and 3) module for tokenizing Icelandic text. It converts Python text strings to streams of token objects, where each token object is a separate word, punctuation sign, number/amount, date, e-mail, URL/URI, etc. It also segments the token stream into sentences, considering corner cases such as abbreviations and dates in the middle of sentences.

The package contains a dictionary of common Icelandic abbreviations, in the file src/tokenizer/Abbrev.conf.

Tokenizer is derived from a corresponding module in the Greynir project (GitHub repository here), by the same author. Note that Tokenizer is licensed under the MIT license while Greynir is licensed under GPLv3.

You might also find the Reynir natural language parser for Icelandic, interesting. The Reynir parser uses Tokenizer on its input.

To install:

$ pip install tokenizer

To use (for Python 3, you can omit the u"" string prefix):

from tokenizer import tokenize, TOK

text = (u"Málinu var vísað til stjórnskipunar- og eftirlitsnefndar "
        u"skv. 3. gr. XVII. kafla laga nr. 10/2007 þann 3. janúar 2010.")

for token in tokenize(text):

        print(u"{0}: '{1}' {2}".format(
                TOK.descr[token.kind],
                token.txt or "-",
                token.val or ""))

Output:

S_BEGIN: '-' (0, None)
WORD: 'Málinu'
WORD: 'var'
WORD: 'vísað'
WORD: 'til'
WORD: 'stjórnskipunar- og eftirlitsnefndar'
WORD: 'skv.' [('samkvæmt', 0, 'fs', 'skst', 'skv.', '-')]
ORDINAL: '3.' 3
WORD: 'gr.' [('grein', 0, 'kvk', 'skst', 'gr.', '-')]
ORDINAL: 'XVII.' 17
WORD: 'kafla'
WORD: 'laga'
WORD: 'nr.' [('númer', 0, 'hk', 'skst', 'nr.', '-')]
NUMBER: '10' (10, None, None)
PUNCTUATION: '/' 4
YEAR: '2007' 2007
WORD: 'þann'
DATE: '3. janúar 2010' (2010, 1, 3)
PUNCTUATION: '.' 3
S_END: '-'

Note the following:

• Sentences are delimited by TOK.S_BEGIN and TOK.S_END tokens.

• Composite words, such as stjórnskipunar- og eftirlitsnefndar, are coalesced into one token.

• Well-known abbreviations are recognized and their full expansion is available in the token.val field.

• Ordinal numbers (3., XVII.) are recognized and their value (3, 17) is available in the token.val field.

• Dates, years and times are recognized and the respective year, month, day, hour, minute and second values are included as a tuple in token.val.

• Numbers, both integer and real, are recognized and their value is available in the token.val field.

The tokenize() function

To tokenize a text string, call tokenizer.tokenize(text). This function returns a Python generator of token objects. Each token object is a simple namedtuple with three fields: (kind, txt, val) (see below).

The tokenizer.tokenize() function is typically called in a for loop:

for token in tokenizer.tokenize(mystring):
        kind, txt, val = token
        if kind == tokenizer.TOK.WORD:
                # Do something with word tokens
                pass
        else:
                # Do something else
                pass

Alternatively, create a token list from the returned generator:

token_list = list(tokenizer.tokenize(mystring))

The token object

Each token is represented by a namedtuple with three fields: (kind, txt, val).

The kind field contains one of the following integer constants, defined within the TOK class:

PUNCTUATION = 1
TIME = 2
DATE = 3
YEAR = 4
NUMBER = 5
WORD = 6
TELNO = 7
PERCENT = 8
URL = 9
ORDINAL = 10
TIMESTAMP = 11
CURRENCY = 12       # Not used
AMOUNT = 13
PERSON = 14         # Not used
EMAIL = 15
ENTITY = 16         # Not used
UNKNOWN = 17

S_BEGIN = 11001     # Sentence begin
S_END = 11002       # Sentence end

To obtain a descriptive text for a token kind, use TOK.descr[token.kind] (see example above).

The txt field contains the original source text for the token.

In the case of abbreviations that end a sentence, the final period ‘.’ is a separate token, and it is consequently omitted from the abbreviation token’s txt field. A sentence ending in o.s.frv. will thus end with two tokens, the next-to-last one being the tuple (TOK.WORD, "o.s.frv", "og svo framvegis") - note the omitted period in the txt field - and the last one being (TOK.PUNCTUATION, ".", 3) (the 3 is explained below).

The val field contains auxiliary information, corresponding to the token kind, as follows:

• For TOK.PUNCTUATION, the val field specifies the whitespace normally found around the symbol in question:

  TP_LEFT = 1   # Whitespace to the left
  TP_CENTER = 2 # Whitespace to the left and right
  TP_RIGHT = 3  # Whitespace to the right
  TP_NONE = 4   # No whitespace

• For TOK.TIME, the val field contains an (hour, minute, second) tuple.

• For TOK.DATE, the val field contains a (year, month, day) tuple (all 1-based).

• For TOK.YEAR, the val field contains the year as an integer.

• For TOK.NUMBER, the val field contains a tuple (number, None, None). (The two empty fields are included for compatibility with Greynir.)

• For TOK.WORD, the val field contains the full expansion of an abbreviation, as a list containing a single tuple, or None if the word is not abbreviated.

• For TOK.PERCENT, the val field contains a tuple of (percentage, None, None).

• For TOK.ORDINAL, the val field contains the ordinal value as an integer.

• For TOK.TIMESTAMP, the val field contains a (year, month, day, hour, minute, second) tuple.

• For TOK.AMOUNT, the val field contains an (amount, currency, None, None) tuple. The amount is a float, and the currency is an ISO currency code, i.e. “USD” for dollars ($ sign) or “EUR” for euros (€ sign). (The two empty fields are included for compatibility with Greynir.)

The correct_spaces() function

Tokenizer also contains the utility function tokenizer.correct_spaces(text). This function returns a string after splitting it up and re-joining it with correct whitespace around punctuation tokens. Example:

>>> tokenizer.correct_spaces("Frétt \n  dagsins:Jón\t ,Friðgeir og Páll ! 100  /  2  =   50")
'Frétt dagsins: Jón, Friðgeir og Páll! 100/2 = 50'

The Abbrev.conf file

Abbreviations recognized by Tokenizer are defined in the Abbrev.conf file, found in the src/tokenizer/ directory. This is a text file with abbreviations, their definitions and explanatory comments. The file is loaded into memory during the first call to tokenizer.tokenize() within a process.

Development installation

To install Tokenizer in development mode, where you can easily modify the source files (assuming you have git available):

$ git clone https://github.com/vthorsteinsson/Tokenizer
$ cd Tokenizer
$ # [ Activate your virtualenv here, if you have one ]
$ python setup.py develop

To run the built-in tests, install pytest, cd to your Tokenizer subdirectory (and optionally activate your virtualenv), then run:

$ python -m pytest