tkitAutoMask
自动构建掩码
pip install tkitAutoMask
from tkitAutoMask import autoMask
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("uer/chinese_roberta_L-2_H-128")
# dir(tokenizer)
tomask = autoMask(
# transformer,
mask_token_id = tokenizer.mask_token_id, # the token id reserved for masking
pad_token_id = tokenizer.pad_token_id, # the token id for padding
mask_prob = 0.05, # masking probability for masked language modeling
replace_prob = 0.90, # ~10% probability that token will not be masked, but included in loss, as detailed in the epaper
mask_ignore_token_ids = [tokenizer.cls_token_id,tokenizer.eos_token_id] # other tokens to exclude from masking, include the [cls] and [sep] here
)
详细参考
dev.md
1 maintainer
Metadata
1 maintainer
Release files for tkitAutoMask 0.0.0.116337015
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| tkitAutoMask-0.0.0.116337015.tar.gz | 8.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| tkitAutoMask-0.0.0.116337015-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 15.8 kB
Release files / tkitAutoMask-0.0.0.116337015.tar.gz
| Download URL | tkitAutoMask-0.0.0.116337015.tar.gz |
|---|---|
| Size | 8.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1008c04254af13614e529d8d0bd3b67c09e08d453829e9edba80a5240f6eff32
|
|
BLAKE2b-256 checksum How to use checksums |
d3cf83ec6655c7df0090f07cbe74c612ea4f2454323eb6ea2551304fee597f02
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
|
Release files / tkitAutoMask-0.0.0.116337015-py3-none-any.whl
| Download URL | tkitAutoMask-0.0.0.116337015-py3-none-any.whl |
|---|---|
| Size | 7.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e788d55ee683397fa80d5e3091332e263ddf6ba9cab68e6b2fa0e21cdc5d48b3
|
|
BLAKE2b-256 checksum How to use checksums |
a22bb63ee729bf3a6e074a9a8deed7f4985b17c0434ab5c83bd28e7011a8132a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.12
|
