Sentence embedding evaluation for German
Project description
sentence-embedding-evaluation-german
Sentence embedding evaluation for German.
This library is inspired by SentEval but focuses on German language downstream tasks.
Downstream tasks
The available downstream tasks are listed in the table below. If you that think that a dataset is missing and should be added, please open an issue.
task | type | text type | lang | #train | #test | target | info |
---|---|---|---|---|---|---|---|
TOXIC | ๐ฟ toxic comments | facebook comments | de-DE | 3244 | 944 | binary {0,1} | GermEval 2021, comments subtask 1, ๐ ๐ |
ENGAGE | ๐ค engaging comments | facebook comments | de-DE | 3244 | 944 | binary {0,1} | GermEval 2021, comments subtask 2, ๐ ๐ |
FCLAIM | โ๏ธ fact-claiming comments | facebook comments | de-DE | 3244 | 944 | binary {0,1} | GermEval 2021, comments subtask 3, ๐ ๐ |
VMWE | โ๏ธ verbal idioms | newspaper | de-DE | 6652 | 1447 | binary (figuratively, literally) | GermEval 2021, verbal idioms, ๐ ๐ |
OL19-A | ๐ฟ offensive language | tweets | de-DE | 3980 | 3031 | binary {0,1} | GermEval 2018, ๐ ๐ |
OL19-B | ๐ฟ offensive language, fine-grained | tweets | de-DE | 3980 | 3031 | 4 catg. (profanity, insult, abuse, oth.) | GermEval 2018, ๐ ๐ |
OL19-C | ๐ฟ explicit vs. implicit offense | tweets | de-DE | 1921 | 930 | binary (explicit, implicit) | GermEval 2018, ๐ ๐ |
OL18-A | ๐ฟ offensive language | tweets | de-DE | 5009 | 3398 | binary {0,1} | GermEval 2018, ๐ |
OL18-B | ๐ฟ offensive language, fine-grained | tweets | de-DE | 5009 | 3398 | 4 catg. (profanity, insult, abuse, oth.) | GermEval 2018, ๐ |
ABSD-1 | ๐คท relevance classification | 'Deutsche Bahn' customer feedback | de-DE | 19432 | 2555 | binary | GermEval 2017, ๐ |
ABSD-2 | ๐๐๐ก sentiment analysis | 'Deutsche Bahn' customer feedback | de-DE | 19432 | 2555 | 3 catg. (pos., neg., neutral) | GermEval 2017, ๐ |
ABSD-3 | ๐ค๏ธ aspect categories | 'Deutsche Bahn' customer feedback | de-DE | 19432 | 2555 | 20 catg. | GermEval 2017, ๐ |
MIO-S | ๐๐๐ก sentiment analysis | 'Der Standard' newspaper article web comments | de-AT | 1799 | 1800 | 3 catg. | One Million Posts Corpus, ๐ |
MIO-O | ๐คท off-topic comments | 'Der Standard' newspaper article web comments | de-AT | 1799 | 1800 | binary | One Million Posts Corpus, ๐ |
MIO-I | ๐ฟ inappropriate comments | 'Der Standard' newspaper article web comments | de-AT | 1799 | 1800 | binary | One Million Posts Corpus, ๐ |
MIO-D | ๐ฟ discriminating comments | 'Der Standard' newspaper article web comments | de-AT | 1799 | 1800 | binary | One Million Posts Corpus, ๐ |
MIO-F | ๐ก feedback comments | 'Der Standard' newspaper article web comments | de-AT | 3019 | 3019 | binary | One Million Posts Corpus, ๐ |
MIO-P | โ๏ธ personal story comments | 'Der Standard' newspaper article web comments | de-AT | 4668 | 4668 | binary | One Million Posts Corpus, ๐ |
MIO-A | โด๏ธ argumentative comments | 'Der Standard' newspaper article web comments | de-AT | 1799 | 1800 | binary | One Million Posts Corpus, ๐ |
SBCH-S | ๐๐๐ก sentiment analysis | 'chatmania' app comments, only comments labelled as Swiss German are included | gsw | 394 | 394 | 3 catg. | SB-CH Corpus, ๐ |
SBCH-L | โฐ๏ธ dialect classification | 'chatmania' app comments | gsw | 748 | 748 | binary | SB-CH Corpus, ๐ |
ARCHI | โฐ๏ธ dialect classification | gsw | 18809 | 4743 | 4 catg. | ArchiMob, ๐ ๐ | |
LSDC | ๐ dialect classification | nds | 74140 | 8602 | 14 catg. | Lower Saxon Dialect Classification, ๐ ๐ |
Download datasets
bash download-datasets.sh
Usage example
Import the required Python packages.
from typing import List
import sentence_embedding_evaluation_german as seeg
import torch
Step (1) Load your pretrained model
In the following example, we generate a random embedding matrix for demonstration purposes.
# (1) Instantiate an embedding model
emb_dim = 512
vocab_sz = 128
emb = torch.randn((vocab_sz, emb_dim), requires_grad=False)
emb = torch.nn.Embedding.from_pretrained(emb)
assert emb.weight.requires_grad == False
Step (2) Specify your preprocessor
function
You need to specify your own preprocessing routine.
The preprocessor
function must convert a list of strings batch
(List[str]
)
into a list of feature vectors, or resp. a list of sentence embeddings (List[List[float]]
).
In the following example, we generate some sort of token IDs, retrieve the vectors from our random matrix, and average these to feature vectors for demonstration purposes.
# (2) Specify the preprocessing
def preprocesser(batch: List[str], params: dict=None) -> List[List[float]]:
""" Specify your embedding or pretrained encoder here
Paramters:
----------
batch : List[str]
A list of sentence as string
params : dict
The params dictionary
Returns:
--------
List[List[float]]
A list of embedding vectors
"""
features = []
for sent in batch:
try:
ids = torch.tensor([ord(c) % 128 for c in sent])
except:
print(sent)
h = emb(ids)
features.append(h.mean(axis=0))
features = torch.stack(features, dim=0)
return features
Step (3) Training settings
We suggest to train a final layer with bias term ('bias':True
),
on a loss function weighted by the class frequency ('balanced':True
),
a batch size of 128, an over 500 epochs without early stopping.
# (3) Training settings
params = {
'datafolder': './datasets',
'bias': True,
'balanced': True,
'batch_size': 128,
'num_epochs': 500,
# 'early_stopping': True,
# 'split_ratio': 0.2, # if early_stopping=True
# 'patience': 5, # if early_stopping=True
}
Step (4) Downstream tasks
We suggest to run the following downstream tasks.
FCLAIM
flags comments that requires manual fact-checking because these contain reasoning, arguments or claims that might be false.
VMWE
differentiates texts with figurative or literal multi-word expressions.
OL19-C
distincts between explicit and implicit offensive language.
ABSD-2
is a sentiment analysis dataset with customer reviews.
These four dataset so far can be assumed to be Standard German from Germany (de-DE).
MIO-P
flags Austrian German (de-AT) comments if these contain personal stories.
ARCHI
is a Swiss (gsw), and LSDC
a Lower German (nds) dialect identification task.
# (4) Specify downstream tasks
downstream_tasks = ['FCLAIM', 'VMWE', 'OL19-C', 'ABSD-2', 'MIO-P', 'ARCHI', 'LSDC']
Step (5) Run the experiments
Finally, start the evaluation. The suggested downstream tasks (step 4) with 500 epochs (step 3) might requires 10-40 minutes but it's highly dependent on your computing resources. So grab a โ or ๐ต.
# (5) Run experiments
results = seeg.evaluate(downstream_tasks, preprocesser, **params)
Appendix
Installation & Downloads
The sentence-embedding-evaluation-german
git repo is available as PyPi package
pip install sentence-embedding-evaluation-german
pip install git+ssh://git@github.com/ulf1/sentence-embedding-evaluation-german.git
You need to download the datasets as well.
If you run the following code, the datasets should be in a folder ./datasets
.
wget -q "https://raw.githubusercontent.com/ulf1/sentence-embedding-evaluation-german/main/download-datasets.sh" -O download-datasets.sh
bash download-datasets.sh
Development work for this package
Install a virtual environment
python3.7 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt --no-cache-dir
pip install -r requirements-dev.txt --no-cache-dir
pip install -r requirements-demo.txt --no-cache-dir
(If your git repo is stored in a folder with whitespaces, then don't use the subfolder .venv
. Use an absolute path without whitespaces.)
Python commands
- Jupyter for the examples:
jupyter lab
- Check syntax:
flake8 --ignore=F401 --exclude=$(grep -v '^#' .gitignore | xargs | sed -e 's/ /,/g')
Publish package
pandoc README.md --from markdown --to rst -s -o README.rst
python setup.py sdist
twine upload -r pypi dist/*
Clean up
find . -type f -name "*.pyc" | xargs rm
find . -type d -name "__pycache__" | xargs rm -r
rm -r .pytest_cache
rm -r .venv
New Dataset recommendation
If you want to recommend another or a new dataset, please open an issue.
Troubleshooting
If you have troubles to get this package running, please open an issue for support.
Contributing
Please contribute using Github Flow. Create a branch, add commits, and open a pull request.
Citation
If you want to use this package in a research paper, please open an issue because we have not yet decided how to make this package citable. You should at least mention the PyPi version in your paper to ensure reproducibility.
You certainly need to cite the actual evaluation datasets in your paper. Please check the hyperlinks in the info column of the table above.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Hashes for sentence-embedding-evaluation-german-0.1.4.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | baa064600d2fb680271d41190e8e421bc775df85941f0deeeda58b1b1059a49b |
|
MD5 | ccbefd1700fe707361641d6f560aa4d3 |
|
BLAKE2b-256 | 8840c1d56c4a621c8d083706b20103a1aca3e28639fced50c382e27e2120ebfe |