Modified Kneser-ney Smoothing Language Model
Project description
== knlm ==
Modified Kneser-Ney smoothing language model module for Python
=== Installation ===
pip install knlm
pip3 install knlm
=== Example ===
from knlm import KneserNey
mode = 'build'
if mode == 'build':
# build model from corpus text. order = 3, word size = 4 byte
mdl = KneserNey(3, 4)
for line in open('corpus.txt', encoding='utf-8'):
mdl.train(line.lower().strip().split())
mdl.optimize()
mdl.save('language.model')
else:
# load model from binary file
mdl = KneserNey.load('language.model')
print('Loaded')
print('Order: %d, Vocab Size: %d, Vocab Width: %d' % (mdl.order, mdl.vocabs, mdl._wsize))
# evaluate sentence score
print(mdl.evaluateSent('I love kiwi .'.split()))
print(mdl.evaluateSent('ego kiwi amo .'.split()))
# evaluate scores for each word
print(mdl.evaluateEachWord('I love kiwi .'.split()))
print(mdl.evaluateEachWord('ego kiwi amo .'.split()))
Modified Kneser-Ney smoothing language model module for Python
=== Installation ===
pip install knlm
pip3 install knlm
=== Example ===
from knlm import KneserNey
mode = 'build'
if mode == 'build':
# build model from corpus text. order = 3, word size = 4 byte
mdl = KneserNey(3, 4)
for line in open('corpus.txt', encoding='utf-8'):
mdl.train(line.lower().strip().split())
mdl.optimize()
mdl.save('language.model')
else:
# load model from binary file
mdl = KneserNey.load('language.model')
print('Loaded')
print('Order: %d, Vocab Size: %d, Vocab Width: %d' % (mdl.order, mdl.vocabs, mdl._wsize))
# evaluate sentence score
print(mdl.evaluateSent('I love kiwi .'.split()))
print(mdl.evaluateSent('ego kiwi amo .'.split()))
# evaluate scores for each word
print(mdl.evaluateEachWord('I love kiwi .'.split()))
print(mdl.evaluateEachWord('ego kiwi amo .'.split()))
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
knlm-0.1.1.tar.gz
(10.8 kB
view hashes)
Built Distributions
knlm-0.1.1-cp37-cp37m-win_amd64.whl
(186.2 kB
view hashes)
knlm-0.1.1-cp37-cp37m-win32.whl
(162.8 kB
view hashes)
knlm-0.1.1-cp36-cp36m-win_amd64.whl
(186.2 kB
view hashes)
knlm-0.1.1-cp36-cp36m-win32.whl
(162.8 kB
view hashes)
knlm-0.1.1-cp35-cp35m-win_amd64.whl
(186.2 kB
view hashes)
knlm-0.1.1-cp35-cp35m-win32.whl
(162.8 kB
view hashes)
Close
Hashes for knlm-0.1.1-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9bdef5b0b9361057ad6766a06fe839957ad9a4885736cfa5d6317f2d984057c5 |
|
MD5 | 39fc93aa69f22520574ee96d856fbeee |
|
BLAKE2b-256 | 7347d9ca7a56d08e8d62c2ea0ec46f4a3548656a0ce2a9bb58612b982a954dcc |
Close
Hashes for knlm-0.1.1-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 161fdbbc8c7a16e19309c38b00ecd07310777e74601d2d5e5b974fa4979b0261 |
|
MD5 | 8c3e2286425bff668d5322632c580855 |
|
BLAKE2b-256 | cde6f64501ea72c13611c989c4f201b387853428705dfe7760e7f4cd678eba68 |
Close
Hashes for knlm-0.1.1-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7ce825871ef27b6abfbf5546687f6d6f179e2b2bb45b457e60803d23f148e539 |
|
MD5 | eefcc0f065fd6697319378c578562afd |
|
BLAKE2b-256 | 17e71a6e52b4ced024bf9138886a0cdcbf4c738c24bc3fd3d3480ce10fe48148 |