unwanted-content-detector 0.1.0

A library to detect undesired, unbranded, or harmful content

unwanted_content_detector

A library to detect undesired, unbranded, or harmful content

Usage

In python:

pip install unwanted-content-detector

Minimal

from unwanted_content_detector import Detector
detector = Detector(models=['hatefult_content_generic_distil_bert_finetuned'])
if detector.is_unwanted('content generated by llm'):
    print("Wont continue")

With spark

spark_df.with_column('is_rejected', lambda row: detector.is_unwanted)

In the terminal

./cli.py inference infer 'text to be validated'

Training

Fine tunning

from unwanted_content_detector import Detector
model = Detector({'data_source': df}).train()

./cli.py train

Target Architecture / Features

• multiple Swappable models
• multiple evaluation datasets
• possibility of configuring a custom personal dataset to fine tune
• Single performance evaluation criteria

Use cases it could be applied to

• detecting the generation of harmful content from LLMs
• preventing harmful prompts to be injected into LLMs
• using it as a validator of content being generated according to the brand guidelines

Liability

This tool aims to help you to detect harmful content but it is not meant to be used as the final decision maker alone.