sentence-embedding-evaluation-german

Sentence embedding evaluation for German.

This library is inspired by SentEval but focuses on German language downstream tasks.

Downstream tasks

The available downstream tasks are listed in the table below. If you that think that a dataset is missing and should be added, please open an issue.

Download datasets

bash download-datasets.sh

Check if files were actually downloaded

find ./datasets/**/ -exec ls -lh {} \;

Usage example

Import the required Python packages.

from typing import List
import sentence_embedding_evaluation_german as seeg
import torch

Step (1) Load your pretrained model

In the following example, we generate a random embedding matrix for demonstration purposes.

# (1) Instantiate an embedding model
emb_dim = 512
vocab_sz = 128
emb = torch.randn((vocab_sz, emb_dim), requires_grad=False)
emb = torch.nn.Embedding.from_pretrained(emb)
assert emb.weight.requires_grad == False

Step (2) Specify your preprocessor function

You need to specify your own preprocessing routine. The preprocessor function must convert a list of strings batch (List[str]) into a list of feature vectors, or resp. a list of sentence embeddings (List[List[float]]). In the following example, we generate some sort of token IDs, retrieve the vectors from our random matrix, and average these to feature vectors for demonstration purposes.

# (2) Specify the preprocessing
def preprocesser(batch: List[str], params: dict=None) -> List[List[float]]:
    """ Specify your embedding or pretrained encoder here
    Paramters:
    ----------
    batch : List[str]
        A list of sentence as string
    params : dict
        The params dictionary
    Returns:
    --------
    List[List[float]]
        A list of embedding vectors
    """
    features = []
    for sent in batch:
        try:
            ids = torch.tensor([ord(c) % 128 for c in sent])
        except:
            print(sent)
        h = emb(ids)
        features.append(h.mean(axis=0))
    features = torch.stack(features, dim=0)
    return features

Step (3) Training settings

We suggest to train a final layer with bias term ('bias':True), on a loss function weighted by the class frequency ('balanced':True), a batch size of 128, an over 500 epochs without early stopping.

# (3) Training settings
params = {
    'datafolder': './datasets',
    'bias': True,
    'balanced': True,
    'batch_size': 128, 
    'num_epochs': 500,
    # 'early_stopping': True,
    # 'split_ratio': 0.2,  # if early_stopping=True
    # 'patience': 5,  # if early_stopping=True
}

Step (4) Downstream tasks

We suggest to run the following downstream tasks. FCLAIM flags comments that requires manual fact-checking because these contain reasoning, arguments or claims that might be false. VMWE differentiates texts with figurative or literal multi-word expressions. OL19-C distincts between explicit and implicit offensive language. ABSD-2 is a sentiment analysis dataset with customer reviews. These four dataset so far can be assumed to be Standard German from Germany (de-DE). MIO-P flags Austrian German (de-AT) comments if these contain personal stories. ARCHI is a Swiss (gsw), and LSDC a Lower German (nds) dialect identification task.

# (4) Specify downstream tasks
downstream_tasks = ['FCLAIM', 'VMWE', 'OL19-C', 'ABSD-2', 'MIO-P', 'ARCHI', 'LSDC']

Step (5) Run the experiments

Finally, start the evaluation. The suggested downstream tasks (step 4) with 500 epochs (step 3) might requires 10-40 minutes but it's highly dependent on your computing resources. So grab a ☕ or 🍵.

# (5) Run experiments
results = seeg.evaluate(downstream_tasks, preprocesser, **params)

Demo notebooks

Start Jupyter

source .venv/bin/activate
jupyter lab

Open an demo notebook

• Generic demo
• deepset example
• fasttext example
• SBert example

Appendix

Installation & Downloads

The sentence-embedding-evaluation-german git repo is available as PyPi package

pip install sentence-embedding-evaluation-german
pip install git+ssh://git@github.com/ulf1/sentence-embedding-evaluation-german.git

You need to download the datasets as well. If you run the following code, the datasets should be in a folder ./datasets.

wget -q "https://raw.githubusercontent.com/ulf1/sentence-embedding-evaluation-german/main/download-datasets.sh" -O download-datasets.sh 
bash download-datasets.sh

Development work for this package

Install a virtual environment

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt --no-cache-dir
pip install -r requirements-dev.txt --no-cache-dir
pip install -r requirements-demo.txt --no-cache-dir

(If your git repo is stored in a folder with whitespaces, then don't use the subfolder .venv. Use an absolute path without whitespaces.)

Install conda environment for GPU

conda install -y pip
conda create -y --name gpu-venv-seeg python=3.9 pip
conda activate gpu-venv-seeg
# install CUDA support
conda install -y cudatoolkit=11.3.1 cudnn=8.3.2 -c conda-forge
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/
pip install torch==1.12.1+cu113 torchvision torchaudio -f https://download.pytorch.org/whl/torch_stable.html
# install other packages
pip install -r requirements.txt --no-cache-dir
pip install -r requirements-dev.txt --no-cache-dir
pip install -r requirements-demo.txt --no-cache-dir
watch -n 0.5 nvidia-smi

Python commands

• Jupyter for the examples: jupyter lab
• Check syntax: flake8 --ignore=F401 --exclude=$(grep -v '^#' .gitignore | xargs | sed -e 's/ /,/g')

Publish package

pandoc README.md --from markdown --to rst -s -o README.rst
python setup.py sdist 
twine upload -r pypi dist/*

Clean up

find . -type f -name "*.pyc" | xargs rm
find . -type d -name "__pycache__" | xargs rm -r
rm -r .pytest_cache
rm -r .venv

New Dataset recommendation

If you want to recommend another or a new dataset, please open an issue.

Troubleshooting

If you have troubles to get this package running, please open an issue for support.

Contributing

Please contribute using Github Flow. Create a branch, add commits, and open a pull request.

Citation

If you want to use this package in a research paper, please open an issue because we have not yet decided how to make this package citable. You should at least mention the PyPi version in your paper to ensure reproducibility.

You certainly need to cite the actual evaluation datasets in your paper. Please check the hyperlinks in the info column of the table above.