Textbook: Universal NLP Datasets
Dependency
av==6.2.0jsonnet==0.14.0opencv_python==4.1.1.26torch==1.3.1torchvision==0.4.2numpy==1.17.4transformers=2.1.1
Download raw datasets
bash fetch.sh
It downloads alphanli, hellaswag, physicaliqa and socialiqa from AWS.
In case you want to use something-something, pelase download the dataset from 20bn's website
Usage
from transformers import BertTokenizer
from textbook import *
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text_renderer = TextRenderer(tokenizer)
anli_tool = BatchTool(tokenizer, max_seq_len=128, source="anli")
anli_dataset = TextDataset(path='data_cache/alphanli/eval.jsonl',
config=ANLIConfiguration(), renderers=[text_renderer])
anli_iter = DataLoader(anli_dataset, batch_size=2, collate_fn=anli_tool.collate_fn)
hellaswag_tool = BatchTool(tokenizer, max_seq_len=128, source="hellaswag")
hellaswag_dataset = TextDataset(path='data_cache/hellaswag/eval.jsonl',
config=HellaswagConfiguration(), renderers=[text_renderer])
hellaswag_iter = DataLoader(hellaswag_dataset, batch_size=2, collate_fn=hellaswag_tool.collate_fn)
siqa_tool = BatchTool(tokenizer, max_seq_len=128, source="siqa")
siqa_dataset = TextDataset(path='data_cache/socialiqa/eval.jsonl',
config=SIQAConfiguration(), renderers=[text_renderer])
siqa_iter = DataLoader(siqa_dataset, batch_size=2, collate_fn=siqa_tool.collate_fn)
piqa_tool = BatchTool(tokenizer, max_seq_len=128, source="piqa")
piqa_dataset = TextDataset(path='data_cache/physicaliqa/eval.jsonl',
config=PIQAConfiguration(), renderers=[text_renderer])
piqa_iter = DataLoader(piqa_dataset, batch_size=2, collate_fn=piqa_tool.collate_fn)
# video_renderer = VideoRenderer(data_dir="data_cache/smthsmth/20bn-something-something-v2")
# smth_tool = BatchTool(tokenizer, max_seq_len=128, source="smth", mlm=True)
# smth_config = SMTHSMTHConfiguration()
# smth_dataset = VideoDataset("data_cache/smthsmth/something-something-v2-validation.json",
# smth_config, [text_renderer, video_renderer])
# smth_iter = DataLoader(smth_dataset, batch_size=2, collate_fn=smth_tool.uncollate_fn)
dataset = MultiTaskDataset([anli_iter, hellaswag_iter, siqa_iter, piqa_iter], shuffle=True)
for batch in tqdm(DataLoader(dataset, batch_size=1, collate_fn=BatchTool.uncollate_fn)):
pass
1 maintainer
Metadata
1 maintainer
Release files for textbook 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| textbook-0.1.5.tar.gz | 8.8 kB | Details |
Release files / textbook-0.1.5.tar.gz
| Download URL | textbook-0.1.5.tar.gz |
|---|---|
| Size | 8.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6718c6bc38245a61c11ceedae53664518a0287fc7abdc012bb8c91dfd4609e0f
|
|
BLAKE2b-256 checksum How to use checksums |
9550df84230f659e7555385585df24ce4219feb7355d31e14b7dcadece56cfbb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.37.0 CPython/3.7.3
|
