Skip to main content

Intent and Slot-filling on the ATIS dataset task.

Model architecture

For this task, a model was trained to jointly predict a sentence's Intent and Slots (entities). Each word is embedded (using pre-defined word vectors) to capture the word's meaning while a character-level bidirection Long Short-Term Memory(LSTM) Network encodes the word's letters to capture its lexical structure.

The word vector and outputs of the character-level bi-LSTM are then fed into the word-level bi-LSTM which predicts the Intent. The second layer feeds into a Conditional Random Fields (CRF) layer to predict the individual slots (entities).

The model is stored in intents_slots/model.py

Word Vectors

The newly released Poincare word embeddings (100 dimensional) were used as they have been reported to better encode the hierarchical relationships inherent between words you can find the word vectors used in word_vectors/poincare.txt

Demo

you can run a demo of the pre-trained model by running intents_slots/demo.py

Training

The model was trained for 50 epochs and stored in the pretrained_models directory

  • pretrained_models/dataset_info contains all the vocabularies used by the model (character, word, intent, entity) and their mappings to numbers for encoding/decoding
  • pretrained_models/model.h5 are the weights to the model

The model's loss during training over the epochs are shown below:

The model's accuracy at predicting intents and entities (slots) over time are shown below:

You can retrain the model by running intents_slots/train.py

Tests

Joint:

Intents only:

Entites only:

to above results can be obtained by running intents_slots/evaluate.py

Improvements:

To improve the robustness of the model to out of vocab words, the training data was lemmatised prior to training and the model was retrained. Numbers were also masked using a placeholder (e.g. *) to avoid out-of-vocab times appearing (e.g. 9:30 may appear in training but not 9:29).

The results were slightly improved given the above tweaks. Precision, Recall and F1 scores improved across the board (for both intents, entities).

Perfect scores were achieved using the validation set!?? data/atis-2-dev.csv

Future Improvements (TO DO):

  • Balance out training data (its clear that the intent ATIS_FLIGHT dominates the training set)

(and 'O' dominates the entity tags)

(or if we discount this as an entity tag - then "to/fromloc.city_name" tags)

  • e.g. this can be achieved by subsampling or artificially perterbing data to generate more samples (e.g. increase training instance by sliding each sentences one,two,three,etc places)

  • Investigate the Intents & Entities which are scoring relatively low F1 scores

e.g. (intents such as ATIS_DAY_NAME, ATIS_MEAL, ATIS_FLIGHT_TIME, etc)

e.g. (entities such as compartment, booking_class, meal_code, etc)

  • Preprocess intent labels with #?
  • Embed unknown words too (if possible) rather than giving them (1)
  • convert word numbers (e.g. "one") into digits
  • improve slot extraction using additional pre-trained Named Entity Recognition (NER)s from various libraries

Metadata

Release files for slize 0.0.21

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for slize 0.0.21
File Size Uploaded Provenance Actions
slize-0.0.21.tar.gz 7.1 kB Without provenance Details for slize-0.0.21.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for slize 0.0.21
File Interpreter ABI Platform Actions
slize-0.0.21-py3-none-any.whl Python 3 none any Details for slize-0.0.21-py3-none-any.whl

Total release size: 17.7 kB

Release files / slize-0.0.21.tar.gz

Download URL slize-0.0.21.tar.gz
Size 7.1 kB
Tags Source
SHA-256 checksum
How to use checksums
062b7180c3e7e700b525b1626de02fc5622075c919b4b85daa2531cbdbf2c2be
BLAKE2b-256 checksum
How to use checksums
814e17dbebe6c538b0107589252d54a32920a05bbd60646326738eae2c997125
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.7.3

Release files / slize-0.0.21-py3-none-any.whl

Download URL slize-0.0.21-py3-none-any.whl
Size 10.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1527c761b7c187fa1a4eeaa7427ce171dfbb72b2956de4e774f8b47f112145a2
BLAKE2b-256 checksum
How to use checksums
928e683d6c6d2981e824c5b423bf10cb2fc5a210be2bf910dc32a315fd23ff2e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/1.14.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.2.0 requests-toolbelt/0.9.1 tqdm/4.35.0 CPython/3.7.3

Security for slize 0.0.21

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page