Modified Kneser-ney Smoothing Language Model
Project description
knlm
Modified Kneser-Ney smoothing language model module for Python
Installation
$ pip install knlm $ pip3 install knlm
Example
from knlm import KneserNey mode = 'build' if mode == 'build': # build model from corpus text. order = 3, word size = 4 byte mdl = KneserNey(3, 4) for line in open('corpus.txt', encoding='utf-8'): mdl.train(line.lower().strip().split()) mdl.optimize() mdl.save('language.model') else: # load model from binary file mdl = KneserNey.load('language.model') print('Loaded') print('Order: %d, Vocab Size: %d, Vocab Width: %d' % (mdl.order, mdl.vocabs, mdl._wsize)) # evaluate sentence score print(mdl.evaluateSent('I love kiwi .'.split())) print(mdl.evaluateSent('ego kiwi amo .'.split())) # evaluate scores for each word print(mdl.evaluateEachWord('I love kiwi .'.split())) print(mdl.evaluateEachWord('ego kiwi amo .'.split()))
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
knlm-0.1.2.tar.gz
(10.8 kB
view details)
Built Distributions
knlm-0.1.2-cp37-cp37m-win_amd64.whl
(185.6 kB
view details)
knlm-0.1.2-cp37-cp37m-win32.whl
(162.4 kB
view details)
knlm-0.1.2-cp36-cp36m-win_amd64.whl
(185.6 kB
view details)
knlm-0.1.2-cp36-cp36m-win32.whl
(162.4 kB
view details)
knlm-0.1.2-cp35-cp35m-win_amd64.whl
(185.6 kB
view details)
knlm-0.1.2-cp35-cp35m-win32.whl
(162.4 kB
view details)
File details
Details for the file knlm-0.1.2.tar.gz
.
File metadata
- Download URL: knlm-0.1.2.tar.gz
- Upload date:
- Size: 10.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: Python-urllib/3.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
b9c931fa4968016c8e031c81c65cd4614aa3dab7f33f3db5694ea35a5ce9ad4f
|
|
MD5 |
7bb002628f52ef5de862acde1b1b60d8
|
|
BLAKE2b-256 |
1c8990d52b8445db078539321daa538361ed6ecd38ed3362ec01259631fedb8d
|
File details
Details for the file knlm-0.1.2-cp37-cp37m-win_amd64.whl
.
File metadata
- Download URL: knlm-0.1.2-cp37-cp37m-win_amd64.whl
- Upload date:
- Size: 185.6 kB
- Tags: CPython 3.7m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: Python-urllib/3.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
748296ad60328472334a60a0a472b5106735400304c836d152b60e003bb64a39
|
|
MD5 |
cc5702eda288c046e99fd51be6e02fe6
|
|
BLAKE2b-256 |
0e9dce9c73fbae1189ce2692435d7847b44e282687e917281eaa7a122d9e9ebe
|
File details
Details for the file knlm-0.1.2-cp37-cp37m-win32.whl
.
File metadata
- Download URL: knlm-0.1.2-cp37-cp37m-win32.whl
- Upload date:
- Size: 162.4 kB
- Tags: CPython 3.7m, Windows x86
- Uploaded using Trusted Publishing? No
- Uploaded via: Python-urllib/3.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
6e08bb53f96d6d39fb50500b928b84e64943554c9b02ba7a691a641151b0afad
|
|
MD5 |
e4429ee6b3c773758072d01448eed533
|
|
BLAKE2b-256 |
1f4789ec6f4a4916130ba786fb5e376353af1491e9b8579834797ce212074e7d
|
File details
Details for the file knlm-0.1.2-cp36-cp36m-win_amd64.whl
.
File metadata
- Download URL: knlm-0.1.2-cp36-cp36m-win_amd64.whl
- Upload date:
- Size: 185.6 kB
- Tags: CPython 3.6m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: Python-urllib/3.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
599a06f68ef58d8b5fcc2098e02f47d1eebf0f6d33bb11b97fbb3a21fa66c51b
|
|
MD5 |
a705b3a6d0e89b888a41c8ee3588bea6
|
|
BLAKE2b-256 |
d1996622b4a66b40b0365e09a89d0f0f92a18a062c004d2cac11e7afcf637933
|
File details
Details for the file knlm-0.1.2-cp36-cp36m-win32.whl
.
File metadata
- Download URL: knlm-0.1.2-cp36-cp36m-win32.whl
- Upload date:
- Size: 162.4 kB
- Tags: CPython 3.6m, Windows x86
- Uploaded using Trusted Publishing? No
- Uploaded via: Python-urllib/3.6
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
5f6cbe1a1681a3e47bef2ffad6f91c07168fa99ce431ded550545d926ce56bb8
|
|
MD5 |
25c3d761c50bcdc7fb2cf3fc4d5d610c
|
|
BLAKE2b-256 |
b0caf51cea3792ce229f7c472553a396a6b7daf49b1eda8aa881c9349fd34a87
|
File details
Details for the file knlm-0.1.2-cp35-cp35m-win_amd64.whl
.
File metadata
- Download URL: knlm-0.1.2-cp35-cp35m-win_amd64.whl
- Upload date:
- Size: 185.6 kB
- Tags: CPython 3.5m, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: Python-urllib/3.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
1f5471db52e5e5ffe2e1d5eb390346d3c0707cb8806d6a14a4955612ae3de472
|
|
MD5 |
d8a8b531741b5fc4734c7f1309ccfee3
|
|
BLAKE2b-256 |
5b44dbc7b3a03055c9338f35604e930a4643478ee67818f4475622d6f7de6a76
|
File details
Details for the file knlm-0.1.2-cp35-cp35m-win32.whl
.
File metadata
- Download URL: knlm-0.1.2-cp35-cp35m-win32.whl
- Upload date:
- Size: 162.4 kB
- Tags: CPython 3.5m, Windows x86
- Uploaded using Trusted Publishing? No
- Uploaded via: Python-urllib/3.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 |
a9b9afa86e5b7340949c0bdc24210853ca7ce7743da60c7a1678416afb95c948
|
|
MD5 |
d7433ad87f961142ebd111e8cce31678
|
|
BLAKE2b-256 |
64ebb8018c0babd575cbd97bf46befed7b1fd44407491950760056ebbf47c80c
|