Lamonpy, Latin POS Tagger & Lemmatizer for Python
Project description
Lamon (LAtin MOrphological tools, pronounced /leɪmən/) is a simple POS tagger & lemmatizer for Latin written in C++ and Lamonpy is a Python package of Lamon. You can easily obtain lemma and tag of each word in given text using Lamonpy.
Getting Started
You can install Lamonpy easily using pip. (https://pypi.org/project/lamonpy/)
$ pip install --upgrade pip $ pip install lamonpy
The supported OS and Python versions are:
Linux (x86-64) with Python >= 3.5
macOS >= 10.13 with Python >= 3.5
Windows 7 or later (x86, x86-64) with Python >= 3.5
Other OS with Python >= 3.5: Compilation from source code required (with c++11 compatible compiler)
Here is a simple example using Lamonpy to analyze Latin texts.
from lamonpy import Lamon lamon = Lamon() score, tagged = lamon.tag('In principio creavit Deus caelum et terram.')[0] print(tagged) # `tagged` is a list of tuples `(start_pos, end_pos, lemma, tag)` # [(0, 2, 'in', 'r--------'), # (3, 12, 'principium', 'n-s---nb-'), # (13, 20, 'creo', 'v3sria---'), # (21, 25, 'deus', 'n-s---mn-'), # (26, 32, 'caelus', 'n-s---ma-'), # (33, 35, 'et', 'c--------'), # (36, 42, 'terra', 'n-s---fa-'), # (42, 43, '.', '---------')]
Tagging Model and Its Accuracy
Lamon’s tagging model is based on BiLSTM network trained with Perseus Latin Dependency Treebanks (4,000 sentences) and self-trained with raw Latin corpora (440,000 sentences) collected by Latina Vivense.
Since there is no available standard for evaluating Latin taggers, we built own test set named vivens of 900 sentences. The result of evaluation is shown below:
vivens (900 sents) |
Perseus (4000 sents) |
|||||
---|---|---|---|---|---|---|
lemma |
tag |
both |
lemma |
tag |
both |
|
Lamon |
94.6 |
83.0 |
81.1 |
89.4 |
80.2 |
76.6 |
Lamon (large) |
94.2 |
83.3 |
81.3 |
89.7 |
81.9 |
78.3 |
Lamon (uv.) |
94.4 |
82.6 |
80.7 |
87.7 |
77.9 |
73.8 |
Backoff |
88.1 |
92.4 |
||||
123 POS |
58.1 |
54.8 |
83.8 |
79.6 |
||
CRF POS |
69.1 |
63.4 |
77.3 |
72.9 |
Lamon : base size (embedding_size:80, hidden_size:160)
Lamon (large) : large size (embedding_size:160, hidden_size:320)
Lamon (uv.) : large size without Perseus’ dataset
Backoff : cltk.lemmatize.latin.backoff.BackoffLatinLemmatizer
CRF POS : cltk.tag.pos.POSTag.tag_crf
For calculating both score of 123 POS and CRF POS, Backoff’s results are used.
Since Lamon and all cltk’s tagger are trained with Perseus’ dataset, the scores for Perseus are not significant for confirming the actual accuracy of each model. Rather, it shows that 123 POS and CRF POS are overfitting to Perseus’s dataset.
Because the size of the vivens dataset is small, the results of this evaluation can be inaccurate. We plan to acquire larger dataset for evaluation and publish the dataset to make more accurate evaluation.
Demo
Larger Models
Due to the package size limit of pypi, the distributed wheel package contains base model only. We provide larger models by Google-drive links.
Large model : https://drive.google.com/file/d/1u8LdvD-zKtrj7kDRs6CjQw74ZG6aT8jS/view?usp=sharing
Large model (unsupervised) : https://drive.google.com/file/d/1nw8LO_1o0O894gXzgQ7Hx5Fyikvy1w2u/view?usp=sharing
You can use these models by passing the model path to Lamon.__init__ as arguments.
from lamonpy import Lamon lamon = Lamon(dict_path='dict.large.bin', tagger_path='tagger.large.bin')
License
Lamonpy is licensed under the terms of MIT License, meaning you can use it for any reasonable purpose and remain in complete ownership of all the documentation you produce.
History
- 0.2.0 (2020-10-16)
[NUM] token for Roman numeral was added.
The accuracy was slightly increased by introducing joint lemma-tag layer.
- 0.1.0 (2020-09-26)
the first version of lamonpy
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for lamonpy-0.2.0-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 229609ce97f00798551e4dd5ea5435fa155ae1c91a1edf4b9ae7c15d6237da07 |
|
MD5 | beb8d7c393c46d71c708dcf9664b9bf9 |
|
BLAKE2b-256 | 0edb07ed601b97d1b5d17d9236bab2fe407307224e214494369a7fde46038a69 |
Hashes for lamonpy-0.2.0-cp38-cp38-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 57bf4600b7626b315a0c78f0b77d8bb723ae8eccd0472a709fce0436f7fe9819 |
|
MD5 | 5912e4c051fc32c5c0072da5fe1b8d5e |
|
BLAKE2b-256 | db6e06ca474925ab895646f371151c160aff2c94da39d397794742a07be3dca4 |
Hashes for lamonpy-0.2.0-cp38-cp38-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aa1f130d49008add80faa7533c6504e7b9cdbf0b5198d9f9bba1c42d7914fc9c |
|
MD5 | b4dec6af00f15cc68788d59e52b1e3af |
|
BLAKE2b-256 | 72092edd23e91209d2e08b12676ca7dc55e38e9dbcb821737b3ff2990828729b |
Hashes for lamonpy-0.2.0-cp38-cp38-macosx_10_14_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c94b5dce1135a7ee7a8bfe4c304ecaadde2baa8640bb725add30cf0d3b065724 |
|
MD5 | 64ab2f8f4c5a51c8f8636ae9f8bdb56d |
|
BLAKE2b-256 | c64de090116c5b876bc2cfd52e852c9f33af42b3331ca1cd8f3cf05171a1825d |
Hashes for lamonpy-0.2.0-cp37-cp37m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6fb274bd5d98f83d41493e8be6a21f2acbb1f82b3b11d1805106b816ea75a06e |
|
MD5 | b5969275abc6a9cf312386ceccf50346 |
|
BLAKE2b-256 | 9f87f4de3c2b51a9098b2501c268a2ff2a71886de4025344fa3022c5a60ec433 |
Hashes for lamonpy-0.2.0-cp37-cp37m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a64dad13a6627de1d9c56309f2979f4a8fe859248cb0988f2f9d2d6f497dc636 |
|
MD5 | cf36926d5166426430a3e89c700a6994 |
|
BLAKE2b-256 | abb7c62d645983aa6401d0fddb3965c2cc77bcea46a4ec54267247bd749fbe5d |
Hashes for lamonpy-0.2.0-cp37-cp37m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec98e0a4ef5a72e4645e90467133a320f22b33b4e5fb586ca46d3b4e73bb8eda |
|
MD5 | be036ba01a1cc64776aacf8a99a1e18d |
|
BLAKE2b-256 | fb56bd6bde972a0d03760aeaa396a57d003bd6963ab328338540a5805e687caf |
Hashes for lamonpy-0.2.0-cp37-cp37m-macosx_10_14_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4e51917d69bd7c8aad204a3ddee94917cf3ae6464789f0da6a3ce8b7b7ac1681 |
|
MD5 | 5303bd7f62fe4471fcfac809a7642885 |
|
BLAKE2b-256 | 32b15f8e64f921ed7ae5ce0e48dfb5f52aa3712c00fb1b84c2f6ad8dbaca4ebd |
Hashes for lamonpy-0.2.0-cp36-cp36m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac9d22b61c46f833423593324e895d9b534c9308b717fcd7ae7c080273fadd98 |
|
MD5 | fa1cadc4c65f97cb78feb7434ab7a100 |
|
BLAKE2b-256 | e3d2ee09becb5b1a9603eefb949e521e3659631a168276cb8cecd888061ff7fb |
Hashes for lamonpy-0.2.0-cp36-cp36m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85f026be66b6d0afd0e13a84d079e811d750aa1cf2a5af031987f43e5a72e4e9 |
|
MD5 | 7ef87fd7aa59dc9fda4209320e8dafe0 |
|
BLAKE2b-256 | d80065bef4c5b264288ee06b25098593e62dcfc258357e8e9a1b4dba6df55c6e |
Hashes for lamonpy-0.2.0-cp36-cp36m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b25538e96e31dd7be9ef6bb49a862a29e58679e0aa7653e88e9b9ae3ae2afa6c |
|
MD5 | 486f64a65d62c8cf304918a3b8e5a396 |
|
BLAKE2b-256 | 79892b428dd3682b4d9f044b85bca5e524b200ec0885b889c4a53e2bc322900b |
Hashes for lamonpy-0.2.0-cp36-cp36m-macosx_10_14_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 62bbd2d55f227e7a9a62fa35246c5900f5926c3db75ea23f560eee2047294004 |
|
MD5 | 30cd5f2649ec77cca7ef5f0ce1984d74 |
|
BLAKE2b-256 | 83547056319348d57da41ca648a262a27c8e6a58a420fedf8dd13a5999bcabdf |
Hashes for lamonpy-0.2.0-cp35-cp35m-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b93e962428ac09a5b1fc3e5569346d6bfd94e063828cfb8e54f8aae53baee006 |
|
MD5 | b0bea2f9930ab2b088fb78ca9c968bbb |
|
BLAKE2b-256 | 19f2b0cdf8b25e219a8a72204f19dcf5be46e2736bcb3be2c5e257469b95fa7e |
Hashes for lamonpy-0.2.0-cp35-cp35m-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f212248ef5860965dfe0608d875ccc3b2f99876f81503955ce4de8fc038b383b |
|
MD5 | 35500aff41f5531125c7677f56b3d5d5 |
|
BLAKE2b-256 | 5d0a007310cc83aae436ef9e63501bc9803c4837a072cf1974a0d260a09df948 |
Hashes for lamonpy-0.2.0-cp35-cp35m-manylinux2010_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 51d5e9192fc8042a5fd4dbcbe94b5052dd7ad0be243c2ee6440cc2dbd0161834 |
|
MD5 | c9116eaa8aa38e7e79be3fe6df47e064 |
|
BLAKE2b-256 | dfa4fb049a37be0b4c9c622aa725b7271ce48c2220e06ee6c1170ed1ac2c4e91 |
Hashes for lamonpy-0.2.0-cp35-cp35m-macosx_10_14_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 415cf956804c2a220c62f91a6e2b330925ad7985b1e0fc72f10d5c414c802d86 |
|
MD5 | f53fa0c1a79becfa2f3c2a5441a583a9 |
|
BLAKE2b-256 | 190b150ef811ff50b834ccd8a78ea15f9c4dedaececb59ea0d7725637a30ca8c |