Multilingual Offensive Language Identification with Transformers
Project description
DeepOffense : Multilingual Offensive Language Identification with Transformers
DeepOffense provides state-of-the-art transformer models for multilingual offensive language identification.
Installation
You first need to install PyTorch. THe recommended PyTorch version is 1.5. Please refer to PyTorch installation page regarding the specific install command for your platform.
When PyTorch has been installed, you can install from source by cloning the repository and running:
git clone https://github.com/TharinduDR/DeepOffense.git
cd DeepOffense
pip install -r requirements.txt
Run the examples
Examples are included in the repository but are not shipped with the library. Please refer the examples directory for the examples. Each directory in the examples folder contains different languages.
Pretrained Models
English offensive language detection pre-trained model trained with XLM-R large model on OffensEval data can be downloaded using this link.
Once downloading it and unzipping it, they can be loaded easily. To see how to begin the training process please refer the examples directory
model = ClassificationModel("xlmroberta", "path", use_cuda=torch.cuda.is_available())
Citation
Please consider citing us if you use the library.
@inproceedings{ranasinghe-etal-2020-multilingual,
title = "Multilingual Offensive Language Identification with Cross-lingual Embeddings",
author = "Ranasinghe, Tharindu and
Zampieri, Marcos,
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
month = nov
year = "2020",
}
Citation for the Malayalam specific paper,
@inproceedings{ranasinghe-etal-2020-wlv,
title={WLV-RIT at HASOC 2020: Offensive Language Identification in Code-switched Texts},
author={Ranasinghe, Tharindu and Zampieri, Marcos},
year={2020},
booktitle={Proceedings of FIRE}
}
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for deepoffense-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 454b85dee73e50e91a2cf9a1dd3291f150d2dcd6d24e2eead9198a7b67b8e1c6 |
|
MD5 | a70a49ba6f1ece7501b8aa92566edcc6 |
|
BLAKE2b-256 | 02450c8c0a7711d7a4b7db44dd8323b66b63542c9afbf5163788a7e2b75cce5d |