VariKN language modeling toolkit
Project description
VariKN language modeling toolkit provides tools for training n-gram language models. Amongst the supported methods are:
Absolute discounting
Kneser-Ney smoothing
Revised Kneser pruning
Kneser-Ney growing
The descriptions of Revised Kneser pruning and Kneser-Ney growing can be found in the paper Vesa Siivola, Teemu Hirsimäki and Sami Virpioja, “On Growing and Pruning Kneser-Ney Smoothed N-Gram Models”, IEEE Transactions on Speech, Audio and Language Processing, 15(5):1617-1624, 2007.
The package provides a accurate pruning for Kneser-Ney smoothed models. Also, it is possible to train a very high-order n-gram models with the growing algorithm. The models can be output to arpa lm format, which is compatible with most common other tools in the field. You can look at the paper Vesa Siivola, Mathias Creutz and Mikko Kurimo: “Morfessor and VariKN machine learning tools for speech and language technology”, Proceedings of the 8th International Conference on Speech Communication and Technology (INTERSPEECH’07), 2007 for guidelines on typical use.
For help on installing the toolkit, read [INSTALL.md](INSTALL.md).
For help on using the built commands, read [commands.html](commands.html).
If it looks like there is interest in using the toolkit but the documentation is inadequate, let me know the main sticking points and I’ll try to do something about it. If you are interested in a well supported toolkit that provides a wide range of functionality, you might want to look at the [srilm toolkit](http://www.speech.sri.com/projects/srilm/).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for varikn-1.2.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 495ab0d2dbfda0b51800798eedf698186ee3b083c66e235851eef245c909e519 |
|
MD5 | 011a9df029c721cd8d91f28cf0bfb5b5 |
|
BLAKE2b-256 | af1bfc4b9d08bbf01ba60a7a62ebfd652a6fba8d6d21b376da659b31f709b601 |
Hashes for varikn-1.2.1-pp39-pypy39_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f125ff6aa5f13500bd9cb47d54151de750cd523f997163c5d319aa49cddfd613 |
|
MD5 | 3405a79fe02e35b14b18f6e6da6b2afc |
|
BLAKE2b-256 | 6f6167d4d65d3aca79fcfb768710a1188e05b96a4ebf41ca26c0143968da4fa9 |
Hashes for varikn-1.2.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a8a6aa951b69eedfb6a5f1bce8c87c0341aeab8d0e04d42c31768230b4a6a5ae |
|
MD5 | af1a89adf02c64618fc05603a393c2cd |
|
BLAKE2b-256 | 477484807905e79cd3c5ee7ab3ab5344844acd87071b43baffe961060ef21dd7 |
Hashes for varikn-1.2.1-pp38-pypy38_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c3821a67072e929f95706616750a50367352ca7162d0cdf24c19f9a97e4dc9f7 |
|
MD5 | dad4766643869de0da618668b94278da |
|
BLAKE2b-256 | cb191bc23add24fd099e213c6c6e75d539a1f158c508f44be76becfe57f01a6c |
Hashes for varikn-1.2.1-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14b6c4f1638a6a98255d2b07e89e22c55ddc111e4071b30e87b64d6735b9a759 |
|
MD5 | 4fd63ae4d2a91c4ba2c8a6cdeb3f063e |
|
BLAKE2b-256 | ef52c47326f71416493c20d23e426846410a3e543cd0c451b114c512c70da701 |
Hashes for varikn-1.2.1-pp37-pypy37_pp73-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 86cc00d08d22332405d7cfa4125f68fa337350b4e418a30903265eb6fc31de3d |
|
MD5 | ad986176f1594c45d68c01e24f2577a5 |
|
BLAKE2b-256 | 1e8c852b7d5068131b2d6bc4693e74b284a5c48ea9e1b01b8578d61e7b1f07fc |
Hashes for varikn-1.2.1-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7bb706432beb005aadd8ae95ac8cbb8350f998a0a283a1aa5012f4421de20aaf |
|
MD5 | d82c106f653b415862ab9c4639da30f8 |
|
BLAKE2b-256 | d15ac76564db71cbb637494de8bc8f9c2b3e1097dd1dc1ac0c94aef1a13861b8 |
Hashes for varikn-1.2.1-cp311-cp311-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b7f1d3c51be98b7d1413961ef5fe9ac0f360e80c37db6e84c8dce1d5c6ac89b3 |
|
MD5 | 496cbdc26285e8cf6bcbc284f5472221 |
|
BLAKE2b-256 | 6f4085d0e50dc46d90dd722bd08f4422e45bdea655f6f771c333f1b2e6e4d6a0 |
Hashes for varikn-1.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cb4fa7addce0d815827b9529fccde05ab0691aabe5e9565c60705cb2385b2115 |
|
MD5 | 6524374ba64bfb6d6e492fd85097f244 |
|
BLAKE2b-256 | dfe641f28fb4d3349fbf41d49e09b78d8fe6d660c5873c5b5995e39047562f96 |
Hashes for varikn-1.2.1-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9727eb9e18d3553afaa8b22d1e0a9a57e64ea2994212595e0f783d12acc5b758 |
|
MD5 | 460078839e39d6fc3a2b96230eae8e25 |
|
BLAKE2b-256 | 0c3ae9be57e8912dc2070c11d8b7916d3405b7230131308dfcebd17172f07fe1 |
Hashes for varikn-1.2.1-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 908a1b8f07047c2125e20d903498f06af826e0ec006f7e710c8251f1d3dcbdd2 |
|
MD5 | 15857bb1e66a5c28422a032a7cbb831f |
|
BLAKE2b-256 | bc24bd1ed263a31c19ed180c8eb70d6a4ac793572bf00767ece080a3d702910f |
Hashes for varikn-1.2.1-cp310-cp310-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04dda58a3ca5f83ec89ba7279271c0fda61d26fc661e45e65dd7fea22a600529 |
|
MD5 | 9bf2662888d7cfea9b019395185acf2a |
|
BLAKE2b-256 | 9c798a5abc69ecd6b0173f7d9308952b1ba90ac1dc7760a86260a2476dcb697a |
Hashes for varikn-1.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 25162e88fe48e285156934da5eca806d214be058fe6861ddd7e492ac3b16be6a |
|
MD5 | 42f7513d4552a9be538d62db197e72ed |
|
BLAKE2b-256 | 71a3b45cdd7ff312a9b23626be2a270b7418fa6d5884462aab270b0ce5158720 |
Hashes for varikn-1.2.1-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 095ab3fd7e044f7f29bfc18ba3bc1d1c2b62c97407c33a6cf42c59cd32537749 |
|
MD5 | fd4cfb39a051f4ea42632eacd58de6c5 |
|
BLAKE2b-256 | f1e68e4f17db0211525f3f895f49e7cc4427038bbaaf45aea2ba63f7a36e3b31 |
Hashes for varikn-1.2.1-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 730aa9db0d48578257460675febe79a3c46fede7875b2efea58ffa6cbf5e3fb6 |
|
MD5 | e2efc1e10213ff13d2b05ccbfff432fd |
|
BLAKE2b-256 | c4264fd739ce15718921f783c07c869569a4c2f346f995ba26dd96513f7b8d17 |
Hashes for varikn-1.2.1-cp39-cp39-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 679177f2526b1c8a0efd6dd0d4fe73ee07cbe18dc908e1f7ff4d8b618f466a57 |
|
MD5 | 81479e46f8c3513366f5e01f00538fce |
|
BLAKE2b-256 | f879681a8653d9d4cd4b69b772dec9a2768fd2747921912e71803b5dba09e02f |
Hashes for varikn-1.2.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 42c3f1f0958251204465ac56e7a0c4612ff0b798eac02425b6c7c303a3290072 |
|
MD5 | c4664c2b0d7cf28a785c6b07526bea95 |
|
BLAKE2b-256 | ac6294ade29c3026baab48b0a2d7c5f27a948c4b46b877917545b2a020f83400 |
Hashes for varikn-1.2.1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 114ba547b2fab2035e571815e7316f767bfd7d96e82d02ab1df1fa864e831542 |
|
MD5 | 78ac0320c460a6c195e20ad07e58f277 |
|
BLAKE2b-256 | b15c65fd6e92872758e9448a17c64ac52f32bfbdde12e1789c05df7d83bf513b |
Hashes for varikn-1.2.1-cp38-cp38-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07caf6ca9b90555585acdb1128edf41bf5abb70633c767afad2985f144eeef87 |
|
MD5 | a593b2d7fcc70fd62da1b5072305b3cc |
|
BLAKE2b-256 | 56055b17b62754b7939b11b0ab8ba10d1b0559b25871f23710823838b2d3de83 |
Hashes for varikn-1.2.1-cp38-cp38-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c143119c7994eda8a338f70367cbc87c25f2f425036908ffb5bd5856c30cee45 |
|
MD5 | 5ff1fce1053db9e2cf8ab4a439cb4ce6 |
|
BLAKE2b-256 | cbe8d567e97d574942427702415dc3a14c765aa368153b1f5fcbe5b55fc5afcd |
Hashes for varikn-1.2.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4d6266eab1d1d6046fca18ec32283a093b607e6befe1ab49a3accc3fe5a29d6 |
|
MD5 | ef598fc87af72797caa2be4f483d7411 |
|
BLAKE2b-256 | a76104f50b34ab0b985486ffe1b4d9d8fe52da5c1c118d3d41afa30f3b8a66f0 |
Hashes for varikn-1.2.1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 48c3b8dc229ae8f7adc8aa83a8d097d7f89fd4374fa1cf24cbd05a81a193563b |
|
MD5 | 2413f1fc4f4beb5a90d9f8072eb4f88e |
|
BLAKE2b-256 | 030a0795728c6532e70c720ce54a0cce6840f5caf218006993463d255d79ca3d |
Hashes for varikn-1.2.1-cp37-cp37m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7b78ffb4e9936a0b74c38eab24d484f0c61d7dab5dd40e968e47b61b27469fad |
|
MD5 | 3d397a660fced83f870d1e8b4a785da3 |
|
BLAKE2b-256 | f9890a623de12069d2ae814e52b770378adf1ebb51bf3312d93afe023665faf3 |
Hashes for varikn-1.2.1-cp37-cp37m-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cb6cb43d810c41001b86485803373575138d69cbbad2efd152545d944d50ecad |
|
MD5 | c35752c69a4742f4cc7e90e332474b14 |
|
BLAKE2b-256 | 61eafe337ce6a549a920390610e5b6254a9f78ba16a65924b6d5e2754c437393 |
Hashes for varikn-1.2.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d031dbf6ec77cd460b7625f7f2c332fb9f077e5a804457209e8cb6c138965a62 |
|
MD5 | cf359c16ea60c25951cab60896b4010a |
|
BLAKE2b-256 | 98422ded88f80b29417797664519a50e669062101f9a9dea9928ee00c0ccb468 |
Hashes for varikn-1.2.1-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aed10f8b574dccb607a89feab7fba771c0e185f5c81222003930bd4263fa1aa7 |
|
MD5 | ed13092247f0a693d6a5f4963c244895 |
|
BLAKE2b-256 | ba95640b6f3778a8ea2aef19a9d5a13c86caff04b74d29d2d9766627e507c12e |
Hashes for varikn-1.2.1-cp36-cp36m-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9632574dda4c894782aa9402d5897b9a7ac965d42b2b14e7448a69ac16b0cfc5 |
|
MD5 | 5081b50c832eae0ac86c784d485e696e |
|
BLAKE2b-256 | 30aa25ed13e5fc5047f8064fe1873a6ae84d6eebb06c05e3238ccd84af1ae585 |
Hashes for varikn-1.2.1-cp36-cp36m-musllinux_1_1_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 429366c0c784699055bbb481511fc413ccd3af380f0b385810fa38e8bf48db90 |
|
MD5 | d0bf2753234a0de03506c3b4c2fd8843 |
|
BLAKE2b-256 | 928a0448582f9e2a4e4a261c652427d0bc079c7f212df243c6135d8dfd121570 |
Hashes for varikn-1.2.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c6ae40ad95f8c78407b825a88cf07e8ccbcdf30d9135b7e51343d9a03286b96 |
|
MD5 | 4f3195544781e384119d6bd9d4dfac0c |
|
BLAKE2b-256 | 0565c8b61ea1f0b08e60ab7d68404208e7b119d904a13474084711973952b9b7 |
Hashes for varikn-1.2.1-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 366888544ffc1a8e956f4df55eabac17aa0283a8dba8bd70369e0b54e82e5a6c |
|
MD5 | 339b86828de457c9c2a57c10744b582b |
|
BLAKE2b-256 | b351a3cd2e17e1602053094a1325cafc054c01a0ebeaf276ad3d2437860949d8 |