Keras XLNet
Unofficial implementation of XLNet. Embedding extraction and embedding extract with memory show how to get the outputs of the last transformer layer using pre-trained checkpoints.
Install
pip install keras-xlnet
Usage
Fine-tuning on GLUE
Click the task name to see the demos with base model:
| Task Name | Metrics | Approximate Results on Dev Set |
|---|---|---|
| CoLA | Matthew Corr. | 52 |
| SST-2 | Accuracy | 93 |
| MRPC | Accuracy/F1 | 86/89 |
| STS-B | Pearson Corr. / Spearman Corr. | 86/87 |
| QQP | Accuracy/F1 | 90/86 |
| MNLI | Accuracy | 84/84 |
| QNLI | Accuracy | 86 |
| RTE | Accuracy | 64 |
| WNLI | Accuracy | 56 |
(Only 0s are predicted in WNLI dataset)
Load Pretrained Checkpoints
import os
from keras_xlnet import Tokenizer, load_trained_model_from_checkpoint, ATTENTION_TYPE_BI
checkpoint_path = '.../xlnet_cased_L-24_H-1024_A-16'
tokenizer = Tokenizer(os.path.join(checkpoint_path, 'spiece.model'))
model = load_trained_model_from_checkpoint(
config_path=os.path.join(checkpoint_path, 'xlnet_config.json'),
checkpoint_path=os.path.join(checkpoint_path, 'xlnet_model.ckpt'),
batch_size=16,
memory_len=512,
target_len=128,
in_train_phase=False,
attention_type=ATTENTION_TYPE_BI,
)
model.summary()
Arguments batch_size, memory_len and target_len are maximum sizes used for initialization of memories. The model used for training a language model is returned if in_train_phase is True, otherwise a model used for fine-tuning will be returned.
About I/O
Note that shuffle should be False in either fit or fit_generator if memories are used.
in_train_phase is False
3 inputs:
- IDs of tokens, with shape
(batch_size, target_len). - IDs of segments, with shape
(batch_size, target_len). - Length of memories, with shape
(batch_size, 1).
1 output:
- The feature for each token, with shape
(batch_size, target_len, units).
in_train_phase is True
4 inputs:
- IDs of tokens, with shape
(batch_size, target_len). - IDs of segments, with shape
(batch_size, target_len). - Length of memories, with shape
(batch_size, 1). - Masks of tokens, with shape
(batch_size, target_len).
1 output:
- The probability of each token in each position, with shape
(batch_size, target_len, num_token).
Metadata
Release files for keras-xlnet 0.20.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| keras-xlnet-0.20.0.tar.gz | 20.7 kB | Without provenance | Details for keras-xlnet-0.20.0.tar.gz |
Release files / keras-xlnet-0.20.0.tar.gz
| Download URL | keras-xlnet-0.20.0.tar.gz |
|---|---|
| Size | 20.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
b9e46404b33cd4b89e2281fd97d34acd1483fec5331b3341b1e17175e3b34df9
|
|
BLAKE2b-256 checksum How to use checksums |
a9d2e62648c98d39857d895f85f0dcf58c06ff426c32fca2b1630e620ca66a48
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.4
|
Security for keras-xlnet 0.20.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.