Amharic language tokenizers
Project description
<h1>Amharic Language Tokenizers</h1>
This package contains set of Classes which can be used to encode Amharic language sentences into tokens that could be used by language models. The tokenizers are trained using [Contemporary Amharic Corpus (CACO)](https://www.findke.ovgu.de/findke/en/Research/Data+Sets/Contemporary+Amharic+Corpus+%28CACO%29-p-1142.html) dataset
<h2>Installing using pip</h2> The package can be installed from test.pypi using the following command. <code> pip install -i https://test.pypi.org/simple/ amtokenizers==0.0.1 </code>
<h2>Example usage</h2>
<code> from amtokenizers import AmTokenizer
a = AmTokenizer(10000, 5 , “byte_bpe”) encoded = a.encode(“አበበ በሶ በላ።”, return_tokens=False) print(“encoded”, encoded.tokens) print(“decoded:”, a.decode(encoded.ids)) </code>
<h2>Disclaimer</h2>
This package is highly inspired by Hugging Face’s [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train) tutorial.
# Amharic Langugae Tokenizers
This package contains set of Classes which can be used to encode Amharic language sentences into tokens that could be used by language models. The tokenizers are trained using [Contemporary Amharic Corpus (CACO)](https://www.findke.ovgu.de/findke/en/Research/Data+Sets/Contemporary+Amharic+Corpus+%28CACO%29-p-1142.html) dataset
## Installing
#### Pip installation
`bash pip install -i https://test.pypi.org/simple/ amtokenizers==0.0.5 `
## Sample Code
`python # toolchain (g++, make) is very likely installed along Xcode, just install cmake and pkg-config using homebrew print("Hello world") `
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for amtokenizers-0.0.6-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2709c7e1962f3fcbe1e3e18febb81b3544ff8b7ab4133b36e8a695d47523bf93 |
|
MD5 | 5db5a947466433639f849dc7525a8106 |
|
BLAKE2b-256 | 20f63ce00423bfadd84e837ade7e9b3c77074f605849499ab1570533ddc57bdb |