glimo-hsd 0.1.0

Glimo HSD privacy-preserving CSV pipeline for harmful-speech datasets.

glimo-hsd

Reusable Python package for the Glimo / PrivHSD backend pipeline. The package processes CSV files into privacy-scrubbed, classifier-ready, optionally restated outputs while keeping model weights outside the PyPI wheel.

Install

pip install glimo-hsd
pip install "glimo-hsd[hf]"  # Transformers / torch classifier support

Python API

from glimo_hsd import PipelineConfig, process_csv

result = process_csv(
    "input.csv",
    config=PipelineConfig(
        text_col="text",
        label_col="hs",
        model_id="batinium/glimo-dehatebert-hsd",
        restatement_backend="none",
        final_scrub=True,
    ),
)

print(result.restated_csv)
print(result.audit_csv)

Use classifier_backend="keyword" for offline smoke tests. Production runs should use classifier_backend="hf" with the Hugging Face model repo or a local model directory.

LLM Restatement Backend

The package does not insert instruct tokens, chat-template markers, or model-specific control text. Those are handled by the LLM provider or local runtime.

When restatement_backend is qwen or local-http, Glimo sends an OpenAI-compatible chat-completions request and requires a structured tool call with tool_choice="required". That tool contract is intentional: the restatement step needs exactly one ordered restatement per input row. Providers or runtimes used for restatement must support OpenAI-style tool calling.

CLI

glimo-hsd process input.csv \
  --text-col text \
  --label-col hs \
  --out outputs/run_001 \
  --model-id batinium/glimo-dehatebert-hsd \
  --classifier-backend hf \
  --restatement-backend none \
  --final-scrub

The process command writes deterministic artifacts:

source.csv
scrubbed.csv
dehatebert_predictions.csv
token_importances.csv
restatement_input.csv
restated.csv
final_scrubbed.csv
deviation_audit.csv
manifest.json

Hugging Face Model

The default model ID is:

batinium/glimo-dehatebert-hsd

Export a local checkpoint with:

python scripts/export_dehatebert_for_hf.py \
  --checkpoint path/to/final_model \
  --out dist/hf/glimo-dehatebert-hsd

Then upload with the current Hugging Face CLI:

hf auth whoami
hf repos create batinium/glimo-dehatebert-hsd --type model --exist-ok
hf upload batinium/glimo-dehatebert-hsd dist/hf/glimo-dehatebert-hsd --type model

Safety

Classifier scores and restatement audits are decision-support signals. They are not appropriate for fully automated enforcement without human review. Do not publish raw challenge data, admin uploads, or generated outputs containing private source text.