Intent and Slot-filling on the ATIS dataset task.
Model architecture
For this task, a model was trained to jointly predict a sentence's Intent and Slots (entities). Each word is embedded (using pre-defined word vectors) to capture the word's meaning while a character-level bidirection Long Short-Term Memory(LSTM) Network encodes the word's letters to capture its lexical structure.
The word vector and outputs of the character-level bi-LSTM are then fed into the word-level bi-LSTM which predicts the Intent. The second layer feeds into a Conditional Random Fields (CRF) layer to predict the individual slots (entities).
The model is stored in intents_slots/model.py
Word Vectors
The newly released Poincare word embeddings (100 dimensional) were used as they have been reported to better encode the hierarchical relationships inherent between words
you can find the word vectors used in
word_vectors/poincare.txt
Demo
you can run a demo of the pre-trained model by running intents_slots/demo.py
Training
The model was trained for 50 epochs and stored in the pretrained_models directory
pretrained_models/dataset_infocontains all the vocabularies used by the model (character, word, intent, entity) and their mappings to numbers for encoding/decodingpretrained_models/model.h5are the weights to the model
The model's loss during training over the epochs are shown below:
The model's accuracy at predicting intents and entities (slots) over time are shown below:
You can retrain the model by running intents_slots/train.py
Tests
Joint:
Intents only:
Entites only:
to above results can be obtained by running intents_slots/evaluate.py
Improvements:
To improve the robustness of the model to out of vocab words, the training data was lemmatised prior to training and the model was retrained. Numbers were also masked using a placeholder (e.g. *) to avoid out-of-vocab times appearing (e.g. 9:30 may appear in training but not 9:29).
The results were slightly improved given the above tweaks. Precision, Recall and F1 scores improved across the board (for both intents, entities).
Perfect scores were achieved using the validation set!?? data/atis-2-dev.csv
Future Improvements (TO DO):
- Balance out training data (its clear that the intent ATIS_FLIGHT dominates the training set)
(and 'O' dominates the entity tags)
(or if we discount this as an entity tag - then "to/fromloc.city_name" tags)
-
e.g. this can be achieved by subsampling or artificially perterbing data to generate more samples (e.g. increase training instance by sliding each sentences one,two,three,etc places)
-
Investigate the Intents & Entities which are scoring relatively low F1 scores
e.g. (intents such as ATIS_DAY_NAME, ATIS_MEAL, ATIS_FLIGHT_TIME, etc)
e.g. (entities such as compartment, booking_class, meal_code, etc)
- Preprocess intent labels with #?
- Embed unknown words too (if possible) rather than giving them (1)
- convert word numbers (e.g. "one") into digits
- improve slot extraction using additional pre-trained Named Entity Recognition (NER)s from various libraries
Metadata
Release files for slize 0.0.21
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| slize-0.0.21.tar.gz | 7.1 kB | Without provenance | Details for slize-0.0.21.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| slize-0.0.21-py3-none-any.whl | Python 3 | none | any | Details for slize-0.0.21-py3-none-any.whl |
Total release size: 17.7 kB
Release files / slize-0.0.21.tar.gz
| Download URL | slize-0.0.21.tar.gz |
|---|---|
| Size | 7.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
062b7180c3e7e700b525b1626de02fc5622075c919b4b85daa2531cbdbf2c2be
|
|
BLAKE2b-256 checksum How to use checksums |
814e17dbebe6c538b0107589252d54a32920a05bbd60646326738eae2c997125
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.7.3
|
Release files / slize-0.0.21-py3-none-any.whl
| Download URL | slize-0.0.21-py3-none-any.whl |
|---|---|
| Size | 10.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1527c761b7c187fa1a4eeaa7427ce171dfbb72b2956de4e774f8b47f112145a2
|
|
BLAKE2b-256 checksum How to use checksums |
928e683d6c6d2981e824c5b423bf10cb2fc5a210be2bf910dc32a315fd23ff2e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.7.3
|
Security for slize 0.0.21
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.