ai-sentinel 0.0.2

Helpful utilities for AI Sentinel toxicity detection

AI Sentinel

AI Sentinel is a Python package designed to help developers integrate toxicity analysis into their applications with ease. It provides a simple, unified interface to leverage powerful AI models for detecting and categorizing harmful content in text.

Key Features

• Advanced Toxicity Detection: Comprehensive toxicity detection and classification.
• Multiple LLM Providers: Designed to support various AI model providers (currently Azure OpenAI, and Gemini, with more models being supported in the future).
• Structured Output: Type-safe responses with Pydantic validation.

Getting Started

Installation

ai-sentinel is avalible on PyPI

pip install ai-sentinel

Usage

AI Sentinel is designed to be straightforward to use. You'll primarily interact with the ToxicityGuard and a client specific to your chosen AI model (e.g., AzureOpenAIClient).

from ai_sentinel import AzureOpenAIClient, ToxicityGuard

# Initialize LLM client
client = AzureOpenAIClient(
    api_key="your-api-key",
    model="gpt-4o-mini",
    api_version="2024-02-01",
    azure_endpoint="https://your-resource.openai.azure.com/"
)

# Create toxicity guard
guard = ToxicityGuard(client)

# Analyze text
result = guard.analyze("This is a normal message")

print(f"Is toxic: {result.is_toxic}")
print(f"Confidence: {result.confidence}")
print(f"Categories: {result.categories}")
print(f"Reason: {result.reason}")
print(f"Severity: {result.score}")

Async Usage

Simply call the async analyze function analyze_async and use await with each API call:

import asyncio
from ai_sentinel import AzureOpenAIClient, ToxicityGuard

client = AzureOpenAIClient(
    api_key="your-api-key",
    model="gpt-4o-mini",
    api_version="2024-02-01",
    azure_endpoint="https://your-resource.openai.azure.com/"
)

async def main() -> None:
    guard = ToxicityGuard(client)
    response = await guard.analyze_async("Text to analyze")

    print(f"Is toxic: {result.is_toxic}")
    print(f"Confidence: {result.confidence}")
    print(f"Categories: {result.categories}")
    print(f"Reason: {result.reason}")
    print(f"Severity: {result.score}")

asyncio.run(main())

Supported LLM API Services

ai-sentinel is model agnostic, with support for the following LLM API services:

Provider	Models	Details
Azure OpenAI	GPT-4, GPT-4o, GPT-3.5-turbo	Industry-leading models
Google Gemini	Gemini 2.0 Flash, Gemini 2.5 Pro	Latest Google technology
[Anthropic][anthropic]	To Be Implemented	Will be implemented in the future
Open Source LLMs	To Be Implemented	Will be implemented in the future

Gemini Usage

from ai_sentinel import GeminiClient, ToxicityGuard

# Initialize Gemini client
client = GeminiClient(
    api_key="your-gemini-api-key",
    model="gemini-2.0-flash"
)

# Create and use toxicity guard
guard = ToxicityGuard(client)
result = guard.analyze("Your text here")

Output

In AI Sentinel's ToxicityGuard class, both analyze and analyze_async methods return a ToxicityResult object.

Response Format

class ToxicityResult:
    is_toxic: bool                          # Whether content is toxic
    confidence: float                       # Confidence score that the content is toxic (0.0-1.0)
    categories: List[ToxicityCategories]    # Detected toxicity categories
    reason: str                             # Explanation of the assessment
    score: ToxicityScore                    # Simplified confidence score: "low", "medium", "high"

Example

{
    "is_toxic": True,
    "confidence": 0.90,
    "categories": [
        <ToxicityCategories.THREATS: 'threats'>, 
        <ToxicityCategories.VIOLENCE: 'violence'>
    ],
    "reason": "The phrase 'I will punch you' is a clear and direct threat of physical violence. It expresses an intention to harm another person, categorizing it under threats and violence.",
    "score": <ToxicityScore.HIGH: 'high'>
}

The ToxicityCategories and ToxicityScore enums are available from ai_sentinel.models.

Toxicity Categories

AI Sentinel detects the following toxicity categories:

• Hate Speech: Content attacking individuals/groups based on protected characteristics
• Harassment: Hostile behavior targeting specific individuals
• Threats: Direct or implied threats of violence or harm
• Sexual Content: Inappropriate sexual material
• Self Harm: Content promoting self-injury or suicide
• Violence: Content glorifying or promoting violence
• Bullying: Intimidation or aggressive behavior
• Discrimination: Unfair treatment of specific groups

Configuration

Environment Variables

Tip: Add enviormental variables to a .env file

# Azure OpenAI
AZURE_API_KEY=your-azure-api-key
AZURE_API_VERSION=2024-02-01
AZURE_API_BASE=https://your-resource.openai.azure.com/

# Google Gemini
GEMINI_API_KEY=your-gemini-api-key

Using python-dotenv

from dotenv import load_dotenv
import os

load_dotenv()

client = AzureOpenAIClient(
    api_key=os.getenv("AZURE_API_KEY"),
    model="gpt-4o-mini",
    api_version=os.getenv("AZURE_API_VERSION"),
    azure_endpoint=os.getenv("AZURE_API_BASE")
)

License

This project is licensed under the MIT License - see the LICENSE file for details.