SeqAL

SeqAL is a sequence labeling active learning framework based on Flair.

Installation

SeqAL is available on PyPI:

pip install seqal

SeqAL officially supports Python 3.8+.

Usage

To understand what SeqAL can do, we first introduce the pool-based active learning cycle.

• Step 0: Prepare seed data (a small number of labeled data used for training)
• Step 1: Train the model with seed data
  • Step 2: Predict unlabeled data with the trained model
  • Step 3: Query informative samples based on predictions
  • Step 4: Annotator (Oracle) annotate the selected samples
  • Step 5: Input the new labeled samples to labeled dataset
  • Step 6: Retrain model
• Repeat step2~step6 until the f1 score of the model beyond the threshold or annotation budget is no left

SeqAL can cover all steps except step 0 and step 4. Because there is no 3rd part annotation tool, we can run below script to simulate the active learning cycle.

$python examples/run_al_cycle.py --text_column 0  --tag_column 1 --data_folder ./data/sample_bio --train_file train_seed.txt --dev_file dev.txt --test_file test.txt --pool_file labeled_data_pool.txt --tag_type ner --hidden_size 256 --embeddings glove --use_rnn False --max_epochs 1 --mini_batch_size 32 --learning_rate 0.1 --sampler MaxNormLogProbSampler --query_number 2 --token_based False --iterations 5 --research_mode True

We set research_mode=True. This means that we simulate the active learning cycle. You can also find the script in examples/run_al_cycle.py or examples/active_learning_cycle_research_mode.py. If you want to connect SeqAL with an annotation tool, you can see the script in examples/active_learning_cycle_annotation_mode.py.

You can find more explanations about the parameters in the following tutorials.

Tutorials

We provide a set of quick tutorials to get you started with the library.

• Tutorials on Github Page
• Tutorials on Markown
  • Tutorial 1: Introduction
  • Tutorial 2: Prepare Corpus
  • Tutorial 3: Active Learner Setup
  • Tutorial 4: Prepare Data Pool
  • Tutorial 5: Research and Annotation Mode
  • Tutorial 6: Query Setup
  • Tutorial 7: Annotated Data
  • Tutorial 8: Stopper
  • Tutorial 9: Output Labeled Data
  • Tutorial 10: Performance Recorder
  • Tutorial 11: Multiple Language Support

Performance

Active learning algorithms achieve 97% performance of the best deep model trained on full data using only 30% of the training data on the CoNLL 2003 English dataset. The CPU model can decrease the time cost greatly only sacrificing a little performance.

See performance for more detail about performance and time cost.

Contributing

If you have suggestions for how SeqAL could be improved, or want to report a bug, open an issue! We'd love all and any contributions.

For more, check out the Contributing Guide.

Credits

• Cookiecutter
• browniebroke/cookiecutter-pypackage
• flairNLP/flair
• modal