TakeBlipNer 0.0.1

Named Entity Recognition Package

BiLSTM-CRF on PyTorch

An efficient BiLSTM-CRF implementation that leverages mini-batch operations on multiple GPUs.

Tested on the latest PyTorch Version (0.3.0) and Python 3.5+.

The latest training code utilizes GPU better and provides options for data parallization across multiple GPUs using torch.nn.DataParallel functionality.

Requirements

Install all required packages (other than pytorch) from requirements.txt

pip install -r requirements.txt

Optionally, standalone tensorboard can be installed from https://github.com/dmlc/tensorboard for visualization and plotting capabilities.

Our Training

For local execution run command:

python train.py --input-path files/input/ner_sample.csv --sentence_column MessageProcessed --label_column NER --postag_model_path postag/model.pkl --postag_label_path postag/vocab-label.pkl --save-dir files/output/ --wordembed-path files/input/titan_v2_after_correction_fasttext_window4_mincount20_cbow.kv --epochs 5 --bidirectional --word-dim 300 --separator , 

python train.py --input-path files/input/ner_sample.csv --sentence_column MessageProcessed --label_column NER --postag_model_path postag/model.pkl --postag_label_path postag/vocab-label.pkl --save-dir files/output/ --wordembed-path files/input/titan_v2_after_correction_fasttext_window4_mincount20_cbow.kv --epochs 5 --bidirectional --word-dim 300 --separator , --val --val-path files/input/ner_validation.csv --val-period 1e 

python train.py --input-path files/input/ner_sample.csv --sentence_column MessageProcessed --label_column NER --postag_model_path postag/model.pkl --postag_label_path postag/vocab-label.pkl --save-dir files/output/ --wordembed-path files/input/titan_v2_after_correction_fasttext_window4_mincount20_cbow.kv --epochs 5 --bidirectional --word-dim 300 --separator , --val --val-path files/input/ner_validation.csv --val-period 1e --max-decay-num 2 --max-patience 2 --learning-rate-decay 0.1 --patience-threshold 0.98

For running on Google Colab:

!python train.py --input-path 'files/input/ner_sample.csv' --separator ',' --sentence_column 'MessageProcessed' --label_column 'NER' --postag_model_path postag/model.pkl --postag_label_path postag/vocab-label.pkl --save-dir 'files/output/' --wordembed-path '/content/gdrive/Shared drives/Data & Analytics/D&A Research/TKS/Modelos/Embedding/FastText/Q3 fasttext/titan_v2_after_correction_fasttext_window4_mincount20_cbow.kv' --epochs 5  --word-dim 300  --bidirectional 

!python train.py --input-path 'files/input/ner_sample.csv' --separator ',' --sentence_column 'MessageProcessed' --label_column 'NER' --postag_model_path postag/model.pkl --postag_label_path postag/vocab-label.pkl --save-dir 'files/output/' --wordembed-path '/content/gdrive/Shared drives/Data & Analytics/D&A Research/TKS/Modelos/Embedding/FastText/Q3 fasttext/titan_v2_after_correction_fasttext_window4_mincount20_cbow.kv' --epochs 5 --val --val-path files/input/ner_validation.csv --word-dim 300  --bidirectional --val-period 1e 

!python train.py --input-path 'files/input/ner_sample.csv' --separator ',' --sentence_column 'MessageProcessed' --label_column 'NER' --postag_model_path postag/model.pkl --postag_label_path postag/vocab-label.pkl --save-dir 'files/output/' --wordembed-path '/content/gdrive/Shared drives/Data & Analytics/D&A Research/TKS/Modelos/Embedding/FastText/Q3 fasttext/titan_v2_after_correction_fasttext_window4_mincount20_cbow.kv' --epochs 5 --val --val-path files/input/ner_validation.csv --word-dim 300  --bidirectional --val-period 1e --max-decay-num 2 --max-patience 2 --learning-rate-decay 0.1 --patience-threshold 0.98

Training

Prepare data first. Data must be supplied in one csv file where the first column contain the sentences and the second one the respective labels for that sentence. File might be prepared as follows:

(sample.csv)
MessageProcessed,					Tags
the fat rat sat on a mat,	det adj noun verb prep det noun
the cat sat on a mat,		det noun verb prep det noun
...,						...

Then the above input is provided to train.py using --input-path and the column name for the sentences and the labels using --sentence_column and --label_column.

python train.py --input-path files/input/sample.csv --sentence_column MessageProcessed --label_column Tags ...

You might need to setup several more parameters in order to make it work. Checkout examples/atis for an example of training a simple BiLSTM-CRF model with ATIS dataset. Run python preprocess.py at the example directory to convert to the dataset totrain.py-friendly format, then run

python ../../train.py --config train-atis.yml`

to see a running example. The example configuration assumes that standalone tensorboard is installed (you could turn it off in the configuration file).

For more information on the configurations, check out python train.py --help.

Prediction

For local execution run command for one line prediction:

python predict.py --model-path files/output/model.pkl --input-sentence "frase para prever" --ner-label-vocab files/output/vocab_label.pkl --postag-model-path postag/model.pkl --postag-label-vocab postag/vocab-label.pkl --save-dir files/output/pred.csv --wordembed-path files/input/titan_v2_after_correction_fasttext_window4_mincount20_cbow.kv

For local execution run for batch prediction:

python predict.py --model-path files/output/model.pkl --input-path files/input/sample_predict.csv --sentence-column MessageProcessed --ner-label-vocab files/output/vocab_label.pkl --postag-model-path postag/model.pkl --postag-label-vocab postag/vocab-label.pkl --save-dir files/output/pred.csv --wordembed-path files/input/titan_v2_after_correction_fasttext_window4_mincount20_cbow.kv --separator ,

For Google colab execution run command for batch prediction:

!python predict.py --model-path files/output/model.pkl --input-path files/input/sample_predict.csv --separator "," --sentence-column MessageProcessed --ner-label-vocab files/output/vocab_label.pkl --postag-model-path postag/model.pkl --postag-label-vocab postag/vocab-label.pkl --save-dir files/output/pred.csv --wordembed-path '/content/gdrive/Shared drives/Data & Analytics/D&A Research/TKS/Modelos/Embedding/FastText/Q3 fasttext/titan_v2_after_correction_fasttext_window4_mincount20_cbow.kv'

For Google colab execution run command for one line prediction:

 !python predict.py --model-path files/output/model.pkl --input-sentence "frase para prever" --ner-label-vocab files/output/vocab_label.pkl --postag-model-path postag/model.pkl --postag-label-vocab postag/vocab-label.pkl --save-dir files/output/pred.csv --wordembed-path '/content/gdrive/Shared drives/Data & Analytics/D&A Research/TKS/Modelos/Embedding/FastText/Q3 fasttext/titan_v2_after_correction_fasttext_window4_mincount20_cbow.kv'

Data must be supplied in one csv file with one column which contain the sentences. File might be prepared as follows:

(sample.csv)
MessageProcessed
the fat rat sat on a mat
the cat sat on a mat
...,		

Evaluation

TODO