SEETM (Sinhala-English Equivalent Token Mapper) allows creating equivalent token maps and replace them with a base token to avoid OOV tokens and generate a single feature for all equivalent tokens in a Sinhala-English code-switching dataset in rasa-based conversational AIs.
Features
- Allows mapping multiple equivalent tokens into a base token
- Fully supports rasa 2.8.x projects
- Provides an easy-to-use CLI
- Provides an efficient server-based GUI
- Provides a fully-functional custom whitespace tokenizer
- Fully-supports Sinhala in the GUI
What's Cooking?
- Mapping suggestions in the SEETM server GUI
- Automatically generated mappings
Limitations and Known Issues
- Should manually add the SEETM tokenizer to the rasa pipeline or else the token maps are not taking any effect
- IPA-based suggestions could contain slight changes based on th IPA mapping origin. (SEETM uses CMU)
Resources and References
- CMU Pronunciation Dictionary
- eng-to-ipa pip package (GitHub)
📒 Docs: https://seetm.github.io
📦 PyPi: https://pypi.org/project/seetm/1.1.1/
🪵 Full Changelog: https://github.com/SEETM-NLP/seetm/blob/main/CHANGELOG.md
2 maintainers
Metadata
2 maintainers
Release files for seetm 1.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| seetm-1.1.1.tar.gz | 964.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| seetm-1.1.1-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 2.0 MB
Release files / seetm-1.1.1.tar.gz
| Download URL | seetm-1.1.1.tar.gz |
|---|---|
| Size | 964.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2fc4767d0a197c11f95e7820ff3cbb5a5f5eec076e5b6cd3f4e00ea38a11d8d5
|
|
BLAKE2b-256 checksum How to use checksums |
c2ab269f62ba42b6881bf733db7e6da88af9f57d4ea0869f8b339c7a44254209
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.8.13
|
Release files / seetm-1.1.1-py3-none-any.whl
| Download URL | seetm-1.1.1-py3-none-any.whl |
|---|---|
| Size | 990.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
89585e74dc4b0d44c98654c8709442967861679373af72d3464fb038ea5fec5d
|
|
BLAKE2b-256 checksum How to use checksums |
22263c9787637967becb38301d5f9cffdc67bff1e99740d6699c3c1d8e471ba2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.8.13
|
