Skip to main content

Modified Kneser-ney Smoothing Language Model

Project description

knlm

Modified Kneser-Ney smoothing language model module for Python

Installation

$ pip install knlm
$ pip3 install knlm

Example

from knlm import KneserNey

mode = 'build'
if mode == 'build':
    # build model from corpus text. order = 3, word size = 4 byte
    mdl = KneserNey(3, 4)
    for line in open('corpus.txt', encoding='utf-8'):
        mdl.train(line.lower().strip().split())
    mdl.optimize()
    mdl.save('language.model')
else:
    # load model from binary file
    mdl = KneserNey.load('language.model')
    print('Loaded')
print('Order: %d, Vocab Size: %d, Vocab Width: %d' % (mdl.order, mdl.vocabs, mdl._wsize))

# evaluate sentence score
print(mdl.evaluateSent('I love kiwi .'.split()))
print(mdl.evaluateSent('ego kiwi amo .'.split()))

# evaluate scores for each word
print(mdl.evaluateEachWord('I love kiwi .'.split()))
print(mdl.evaluateEachWord('ego kiwi amo .'.split()))

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

knlm-0.1.2.tar.gz (10.8 kB view details)

Uploaded Source

Built Distributions

knlm-0.1.2-cp37-cp37m-win_amd64.whl (185.6 kB view details)

Uploaded CPython 3.7mWindows x86-64

knlm-0.1.2-cp37-cp37m-win32.whl (162.4 kB view details)

Uploaded CPython 3.7mWindows x86

knlm-0.1.2-cp36-cp36m-win_amd64.whl (185.6 kB view details)

Uploaded CPython 3.6mWindows x86-64

knlm-0.1.2-cp36-cp36m-win32.whl (162.4 kB view details)

Uploaded CPython 3.6mWindows x86

knlm-0.1.2-cp35-cp35m-win_amd64.whl (185.6 kB view details)

Uploaded CPython 3.5mWindows x86-64

knlm-0.1.2-cp35-cp35m-win32.whl (162.4 kB view details)

Uploaded CPython 3.5mWindows x86

File details

Details for the file knlm-0.1.2.tar.gz.

File metadata

  • Download URL: knlm-0.1.2.tar.gz
  • Upload date:
  • Size: 10.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: Python-urllib/3.7

File hashes

Hashes for knlm-0.1.2.tar.gz
Algorithm Hash digest
SHA256 b9c931fa4968016c8e031c81c65cd4614aa3dab7f33f3db5694ea35a5ce9ad4f
MD5 7bb002628f52ef5de862acde1b1b60d8
BLAKE2b-256 1c8990d52b8445db078539321daa538361ed6ecd38ed3362ec01259631fedb8d

See more details on using hashes here.

File details

Details for the file knlm-0.1.2-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: knlm-0.1.2-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 185.6 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: Python-urllib/3.7

File hashes

Hashes for knlm-0.1.2-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 748296ad60328472334a60a0a472b5106735400304c836d152b60e003bb64a39
MD5 cc5702eda288c046e99fd51be6e02fe6
BLAKE2b-256 0e9dce9c73fbae1189ce2692435d7847b44e282687e917281eaa7a122d9e9ebe

See more details on using hashes here.

File details

Details for the file knlm-0.1.2-cp37-cp37m-win32.whl.

File metadata

  • Download URL: knlm-0.1.2-cp37-cp37m-win32.whl
  • Upload date:
  • Size: 162.4 kB
  • Tags: CPython 3.7m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: Python-urllib/3.7

File hashes

Hashes for knlm-0.1.2-cp37-cp37m-win32.whl
Algorithm Hash digest
SHA256 6e08bb53f96d6d39fb50500b928b84e64943554c9b02ba7a691a641151b0afad
MD5 e4429ee6b3c773758072d01448eed533
BLAKE2b-256 1f4789ec6f4a4916130ba786fb5e376353af1491e9b8579834797ce212074e7d

See more details on using hashes here.

File details

Details for the file knlm-0.1.2-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: knlm-0.1.2-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 185.6 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: Python-urllib/3.6

File hashes

Hashes for knlm-0.1.2-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 599a06f68ef58d8b5fcc2098e02f47d1eebf0f6d33bb11b97fbb3a21fa66c51b
MD5 a705b3a6d0e89b888a41c8ee3588bea6
BLAKE2b-256 d1996622b4a66b40b0365e09a89d0f0f92a18a062c004d2cac11e7afcf637933

See more details on using hashes here.

File details

Details for the file knlm-0.1.2-cp36-cp36m-win32.whl.

File metadata

  • Download URL: knlm-0.1.2-cp36-cp36m-win32.whl
  • Upload date:
  • Size: 162.4 kB
  • Tags: CPython 3.6m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: Python-urllib/3.6

File hashes

Hashes for knlm-0.1.2-cp36-cp36m-win32.whl
Algorithm Hash digest
SHA256 5f6cbe1a1681a3e47bef2ffad6f91c07168fa99ce431ded550545d926ce56bb8
MD5 25c3d761c50bcdc7fb2cf3fc4d5d610c
BLAKE2b-256 b0caf51cea3792ce229f7c472553a396a6b7daf49b1eda8aa881c9349fd34a87

See more details on using hashes here.

File details

Details for the file knlm-0.1.2-cp35-cp35m-win_amd64.whl.

File metadata

  • Download URL: knlm-0.1.2-cp35-cp35m-win_amd64.whl
  • Upload date:
  • Size: 185.6 kB
  • Tags: CPython 3.5m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: Python-urllib/3.5

File hashes

Hashes for knlm-0.1.2-cp35-cp35m-win_amd64.whl
Algorithm Hash digest
SHA256 1f5471db52e5e5ffe2e1d5eb390346d3c0707cb8806d6a14a4955612ae3de472
MD5 d8a8b531741b5fc4734c7f1309ccfee3
BLAKE2b-256 5b44dbc7b3a03055c9338f35604e930a4643478ee67818f4475622d6f7de6a76

See more details on using hashes here.

File details

Details for the file knlm-0.1.2-cp35-cp35m-win32.whl.

File metadata

  • Download URL: knlm-0.1.2-cp35-cp35m-win32.whl
  • Upload date:
  • Size: 162.4 kB
  • Tags: CPython 3.5m, Windows x86
  • Uploaded using Trusted Publishing? No
  • Uploaded via: Python-urllib/3.5

File hashes

Hashes for knlm-0.1.2-cp35-cp35m-win32.whl
Algorithm Hash digest
SHA256 a9b9afa86e5b7340949c0bdc24210853ca7ce7743da60c7a1678416afb95c948
MD5 d7433ad87f961142ebd111e8cce31678
BLAKE2b-256 64ebb8018c0babd575cbd97bf46befed7b1fd44407491950760056ebbf47c80c

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page