To computes transition probability of text
Project description
text-trans
It computes a transition probability of a text.
Description
I want to determine if a word was randomly generated. I guess that it can be determined by text transition probabilities trained from correct words.
Vertify
I trained transition probability using almost english words. I computed and compared probability for english words learned at junior high school in Japan, and randomly generated words. From the figure below, it can see that each peak is different.
Install
$ pip install texttrans
Usage
default
Transition probability is computed for English words. I use "words_alpha.txt" of dwyl/english-words to train default probability.
from texttrans.texttrans import TextTrans
p = TextTrans().prob("pen")
print(p)
0.11640052876679541
training
It prepares a text file that lists words, e.g. like below.
hogehoge
piyopiyo
It train text transtion of input text.
from texttrans.texttrans import TextTrans
train_path = "train.txt"
model_path = "model.pki"
tt1 = TextTrans(lang=None)
tt1.train(train_path= train_path, save_path= model_path)
print("p =", tt.prob("hoge"))
It computes the probability according to trained model.
tt2 = TextTrans(model_path=model_path)
print("p =", tt.prob("hoge"))
Licence
Appendix
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for texttrans-1.0.10-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9ff1e7695ce09032cd13268b089455fe7fe2308875b0034e2b1fed958d3bc5eb |
|
MD5 | 1f9169f96683d52441a1a0319bbf2f5a |
|
BLAKE2b-256 | 171aeb29a23d00196cf1ad76f029a4b221974cce941208f20017a591143fa9d6 |