A configuration tool designed to simplify the creation of complete OpenNMT-tf pipelines
AutONMT-tf is a configuration tool designed to simplify the creation of complete OpenNMT-tf pipelines (data loading, preprocessing, training, inference...). It can also be used for other tasks not related to OpenNMT-tf, but there are no built-in modules for other NMT frameworks.
It is still at an early development stage, neither stability nor backward-compatibilty are guaranteed.
AutONMT-tf requires :
- Python 3.7 or above
- OpenNMT-tf 2.20 or above
It is the recommanded (and simplest) installation method :
pip install --upgrade pip pip install AutONMT-tf
You can also install AutONMT-tf directly from source :
git clone https://gitlab.com/mehdidou99/AutONMT-tf.git cd AutONMT-tf pip install --upgrade pip pip install .
Once installed, you can try to run a simple Transformer model pipeline with some preprocessing :
git clone https://gitlab.com/mehdidou99/AutONMT-tf.git cd AutONMT-tf autonmt_cli -v --config examples/pipelines/simple_transformer.yml --pipeline train
The input data used in this example is a toy dataset with several corpora : you can inspect and modify it in examples/data/
Some examples are available in examples/pipelines/:
- simple_encoder.yml: A very simple example showcasing base functionalities of AutONMT-tf
- fren_triple_encoder.yml: A more complex example showcasing the future functionalities of AutONMT-tf, which will allow it to have the flexibility needed for more complex models and pipelines.
AutONMT-tf is used through the
autonmt_cli command line interface.
- Simplest usage :
autonmt_cli --config path/to/pipeline/config/file.yml --pipeline name_of_the_pipeline_to_run
- Key options :
--until step: stops the execution after step step
--use_cache: resumes execution using cache instead of launching the pipeline from the beginning
Each pipeline configuration file is made of the following elements:
- Global configuration
- Pipelines made of pipeline blocks
The simple_transformer example illustrates all of those elements.
The global configuration defines the elements that are used by all the pipelines defined in the file :
- Experiment name
- Custom directories
- Model configuration
- Scripts directory
- Cache directory
Pipelines are the core element of AutONMT-tf. A pipeline is a list of pipeline blocks which each define a specific step of the process : block is applied to a list of corpora; it receives input through input tags and outputs output tags. See Tags to learn more.
AutONMT-tf currently provides the following block types:
- data_query: Loads data : it is usually the first block of a pipeline, and creates the corpora that are later used by the subsequent blocks.
- merging: Used to merge data from several datasets into one new dataset, usually used to merge data for training.
- vocab_building: Builds a vocabulary using the 'onmt-build-vocab' command from OpenNMT-tf.
- splitting: Splits input data into several parts, the intended use is to split train data into train, test and validation sets.
- training: Trains the model using the 'onmt-main' command from OpenNMT-tf.
- script: Executes custom scripts, usually used for experiment-dependent features such as preprocessing, tokenization, score computation...
Modules are currently simply configuration modules allowing blocks to delegate their specific configuration to said module. Their use should be extended in future versions, allowing complete blocks to be defined as modules and allowing external module files in order to allow blocks to be reused in different experiments.
Tags are a core element of AutONMT-tf: they allow pipelines to manipulate data through tags instead of real files, making the pipeline definition much more natural.
The link between abstract tags and underlying real files is handled automatically by AutONMT-tf. For script blocks, the paths to the real files are passed to the script as follows :
script_name input_tag_1_path ... input_tag_N_path output_tag_1_path ... output_tag_M_path
Scripts can thus process data without the script writer needing to know the paths to the real data files.
Some of the generated files are needed by the user, either to be inspected (e.g training data) or to be used in other pipelines. For example, a tokenizer can be trained with the training data, the output of the training being then needed to tokenize test data. Users can retrieve such files through artifacts, by defining correspondancies between Corpora/Tag pairs and custom filenames to which they want to save their files. See simple_transformer for a concrete example.
data_query can load data from several corpora following a query. Only one kind of query is currently supported, here is an example from simple_transformer :
data_query_train: search_root: data/multilingual subpaths: Tatoeba: 'Tatoeba/Tatoeba' GlobalVoices: 'GlobalVoices/GlobalVoices' NewsCommentary: 'NewsCommentary/News-Commentary' TED2013: 'TED2013/TED2013' query: src_lgs: - fr tgt_lgs: - en max_entries: 1000000 options: - try_reversed_lg_pairs
In order to be found by this kind of queries, corpus files must be named with the following convention :
<subpath>.<src>-<tgt>.<src or tgt>
For example, for Tatoeba in the preceding example:
try_reversed_lg_pairs option is active, source and target languages can be reversed in the suffix. e.g :
Other query types will be added in a future version, allowing for less rigid naming conventions.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for AutONMT_tf-0.1.2-py3-none-any.whl