TEA - Translation Engine Architect
Project description
TEA - Translation Engine Architect
A command line tool to create translation engine.
Install
First install pipx then:
pipx install pangeamt-tea
Usage
Step 1: Create a new project
tea new --customer customer --srcLang es --tgtLang en --flavor automotion --version 0.0.1
This command will create the project directory structure:
├── customer_es_en_automotion_0.0.1
│ ├── config.yml
│ └── data
Then enter in the directory
cd customer_es_en_automotion_0.0.1
Step 2: Configuration
Tokenizer
A tokenizer can be applied to source and target
tea tokenizer --src mecab --tgt moses
To list all available tokenizer:
tea tokenizer --list
Truecaser
tea truecaser --src --tgt
BPE
tea bpe -s -t
Processors
tea config processors -s "{processors}"
being processors a list of preprocesses and postprocesses.
Step 3:
Copy some multilingual ressources (.tmx, bilingual files, .af ) into the 'data' directory
Step 4: Run
Clean the data passing the normalizers and validators:
tea workflow clean -n {clean_th} -d
being clean_th the number of threads.
Preprocess the data (split data in train, dev or test, tokenization, BPE):
tea workflow prepare -n {prepare_th} -s 3
being prepare_th the number of threads.
Training model
tea workflow train --gpu 0
Evaluate model
tea workflow eval --step {step} --src file.src --ref file.tgt --log file.log --out file.out --gpu 0
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pangeamt_tea-0.2.26-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2889b2643081ea193a998cc54b2325be9161519621c5cf29d6b64c2f09e9bde9 |
|
MD5 | 9b8f10939344990c5bd08571333522bf |
|
BLAKE2b-256 | 81d1024136d3d483bb3aaf9cfdf3d39367481e75fe54977b9b5231959fd976e4 |