Skip to main content

GLLM Privacy

Description

A library to protect Personal Identifiable Information (PII) in a Generative AI project.

Installation

Prerequisites

Mandatory:

  1. Python 3.11+ — Install here
  2. pip — Install here
  3. uv — Install here

Extras (required only for Artifact Registry installations):

  1. gcloud CLI (for authentication) — Install here, then log in using:
    gcloud auth login
    

Option 1: Install from Artifact Registry

This option requires authentication via the gcloud CLI.

uv pip install \
  --extra-index-url "https://oauth2accesstoken:$(gcloud auth print-access-token)@glsdk.gdplabs.id/gen-ai-internal/simple/" \
  gllm-privacy

Option 2: Install from PyPI

This option requires no authentication. However, it installs the binary wheel version of the package, which is fully usable but does not include source code.

uv pip install gllm-privacy-binary

Local Development Setup

Prerequisites

  1. Python 3.11+ — Install here

  2. pip — Install here

  3. uv — Install here

  4. gcloud CLI — Install here, then log in using:

    gcloud auth login
    
  5. Git — Install here

  6. Access to the GDP Labs SDK GitHub repository


1. Clone Repository

git clone git@github.com:GDP-ADMIN/gl-sdk.git
cd gl-sdk/libs/gllm-privacy

2. Setup Authentication

Set the following environment variables to authenticate with internal package indexes:

export UV_INDEX_GEN_AI_INTERNAL_USERNAME=oauth2accesstoken
export UV_INDEX_GEN_AI_INTERNAL_PASSWORD="$(gcloud auth print-access-token)"
export UV_INDEX_GEN_AI_USERNAME=oauth2accesstoken
export UV_INDEX_GEN_AI_PASSWORD="$(gcloud auth print-access-token)"

3. Quick Setup

Run:

make setup

4. Activate Virtual Environment

source .venv/bin/activate

Local Development Utilities

The following Makefile commands are available for quick operations:

Install uv

make install-uv

Install Pre-Commit

make install-pre-commit

Install Dependencies

make install

Update Dependencies

make update

Run Tests

make test

Usage

from gllm_privacy.pii_detector import TextAnalyzer, TextAnonymizer
from gllm_privacy.pii_detector.constants import Entities
from gllm_privacy.pii_detector.anonymizer import Operation
from asyncio import run

text = """
    contoh nomor ktp 3525011212941001
    repeat nomor ktp 3525011212941001
    contoh email john.doe@example.com
    contoh nomor telepon +628121729819 dan 0812898029384.
    contoh npwp 01.123.456.7-891.234
"""
text_analyzer = TextAnalyzer()
entities = [Entities.EMAIL_ADDRESS, Entities.KTP, Entities.NPWP, Entities.PHONE_NUMBER]

text_anonymizer = TextAnonymizer(text_analyzer)
anonymized_text = run(text_anonymizer.run(text=text, entities=entities))
print(anonymized_text)

deanonymized_text = run(text_anonymizer.run(text=text, entities=entities, operation=Operation.DEANONYMIZE))
print(deanonymized_text)

If you need to detect person, organization, or location entities in text written in Bahasa Indonesia, you can use either TransformersRecognizer or ProsaRemoteRecognizer. To use the TransformersRecognizer, you can use it like this:

from gllm_privacy.pii_detector.recognizer.config import CAHYA_BERT_CONFIGURATION
from gllm_privacy.pii_detector.recognizer.transformers_recognizer import TransformersRecognizer
from gllm_privacy.pii_detector import TextAnalyzer, TextAnonymizer
from gllm_privacy.pii_detector.constants import Entities

# Load the model, if you run it for the first time, it will download the model from the Hugging Face model hub
transformers_recognizer = TransformersRecognizer(
  model_path=CAHYA_BERT_CONFIGURATION.get("DEFAULT_MODEL_PATH"),
  supported_entities=CAHYA_BERT_CONFIGURATION.get("PRESIDIO_SUPPORTED_ENTITIES"),
)
transformers_recognizer.load_transformer(**CAHYA_BERT_CONFIGURATION)
analyzer = TextAnalyzer(additional_recognizers=[transformers_recognizer])

text = "John Doe adalah seorang karyawan PT ABCD yang berlokasi di Jakarta."
text_analyzer = TextAnalyzer(additional_recognizers=[transformers_recognizer])
entities = [Entities.PERSON, Entities.LOCATION]

text_anonymizer = TextAnonymizer(text_analyzer)
anonymized_text = text_anonymizer.anonymize(text=text, entities=entities)
print(anonymized_text)

deanonymized_text = text_anonymizer.deanonymize(text=text)
print(deanonymized_text)

Enhanced TransformersRecognizer with Optimum

The TransformersRecognizer now supports Hugging Face Optimum for improved performance:

  • ONNX Runtime with CUDA: GPU-accelerated inference using ONNX Runtime with CUDA provider
  • ONNX Runtime with CPU: Optimized CPU inference for better performance on laptops/servers
  • Apple Silicon MPS: GPU acceleration on Apple Silicon Macs
  • Auto-detection: Automatically selects the best available backend
  • Fallback compatibility: Works on any hardware with standard transformers

Available Backends:

  • onnx: ONNX Runtime with CPU provider (optimized for NER tasks)
  • cuda: ONNX Runtime with CUDA provider (GPU acceleration)
  • mps: Apple Silicon MPS for GPU acceleration on Mac
  • transformers: Standard transformers as fallback

Configuration Options:

You can configure the backend behavior in your configuration:

config = {
    "USE_OPTIMUM": True,                    # Enable/disable Optimum
    "OPTIMUM_BACKEND": "auto",              # "auto", "onnx", "cuda", "mps", "transformers"
    "OPTIMUM_DEVICE": "auto",               # "auto", "cuda", "cpu", "mps"
    "OPTIMUM_QUANTIZATION": False,          # Enable quantization
    "OPTIMUM_MAX_BATCH_SIZE": 8,           # Max batch size
}

Usage Example:

from gllm_privacy.pii_detector import TextAnalyzer
from gllm_privacy.pii_detector.recognizer.config import CAHYA_BERT_CONFIGURATION
from gllm_privacy.pii_detector.recognizer.transformers_recognizer import TransformersRecognizer

transformers_recognizer = TransformersRecognizer(
    model_path=CAHYA_BERT_CONFIGURATION.get("DEFAULT_MODEL_PATH"),
    supported_entities=CAHYA_BERT_CONFIGURATION.get("PRESIDIO_SUPPORTED_ENTITIES"),
    use_optimum=True
)

transformers_recognizer.load_transformer(**CAHYA_BERT_CONFIGURATION)

pipeline_info = transformers_recognizer.get_pipeline_info()
print(f"Backend: {pipeline_info['backend']}")
print(f"Device: {pipeline_info['device']}")
print(f"Optimizations: {pipeline_info['optimizations']}")

# Use as before
analyzer = TextAnalyzer(additional_recognizers=[transformers_recognizer])

To use the ProsaRemoteRecognizer, you can use it like the following example. Please replace <PROSA_API_URL> and <PROSA_API_KEY> with the valid values.

from gllm_privacy.pii_detector.recognizer.prosa_remote_recognizer import ProsaRemoteRecognizer
from gllm_privacy.pii_detector import TextAnalyzer, TextAnonymizer
from gllm_privacy.pii_detector.constants import Entities

text = "John Doe adalah seorang karyawan PT ABCD yang berlokasi di Jakarta."
prosa_recognizer = ProsaRemoteRecognizer('<PROSA_API_URL>', '<PROSA_API_KEY>')
text_analyzer = TextAnalyzer(additional_recognizers=[prosa_recognizer])
entities = [Entities.PERSON, Entities.LOCATION]

text_anonymizer = TextAnonymizer(text_analyzer)
anonymized_text = text_anonymizer.anonymize(text=text, entities=entities)
print(anonymized_text)

deanonymized_text = text_anonymizer.deanonymize(text=text)
print(deanonymized_text)

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

gllm_privacy_binary-0.4.31-cp313-cp313-win_amd64.whl (555.9 kB view details)

Uploaded CPython 3.13Windows x86-64

gllm_privacy_binary-0.4.31-cp313-cp313-manylinux_2_31_x86_64.whl (876.0 kB view details)

Uploaded CPython 3.13manylinux: glibc 2.31+ x86-64

gllm_privacy_binary-0.4.31-cp313-cp313-macosx_13_0_arm64.whl (606.9 kB view details)

Uploaded CPython 3.13macOS 13.0+ ARM64

gllm_privacy_binary-0.4.31-cp312-cp312-win_amd64.whl (558.9 kB view details)

Uploaded CPython 3.12Windows x86-64

gllm_privacy_binary-0.4.31-cp312-cp312-manylinux_2_31_x86_64.whl (875.5 kB view details)

Uploaded CPython 3.12manylinux: glibc 2.31+ x86-64

gllm_privacy_binary-0.4.31-cp312-cp312-macosx_13_0_arm64.whl (606.4 kB view details)

Uploaded CPython 3.12macOS 13.0+ ARM64

gllm_privacy_binary-0.4.31-cp311-cp311-win_amd64.whl (576.9 kB view details)

Uploaded CPython 3.11Windows x86-64

gllm_privacy_binary-0.4.31-cp311-cp311-manylinux_2_31_x86_64.whl (798.6 kB view details)

Uploaded CPython 3.11manylinux: glibc 2.31+ x86-64

gllm_privacy_binary-0.4.31-cp311-cp311-macosx_13_0_arm64.whl (595.3 kB view details)

Uploaded CPython 3.11macOS 13.0+ ARM64

File details

Details for the file gllm_privacy_binary-0.4.31-cp313-cp313-win_amd64.whl.

File metadata

File hashes

Hashes for gllm_privacy_binary-0.4.31-cp313-cp313-win_amd64.whl
Algorithm Hash digest
SHA256 5aecae674a39ada970af29b623ad662d78a4c2adebe5837c73f43496535378d7
MD5 d2d8768e9689c64d5f2fabe18e50ac7b
BLAKE2b-256 54e24d90f06e7197e1171e4eef2e21441f2fbbf87a1415a65da999f547451be9

See more details on using hashes here.

Provenance

The following attestation bundles were made for gllm_privacy_binary-0.4.31-cp313-cp313-win_amd64.whl:

Publisher: build-binary.yml on GDP-ADMIN/gl-sdk

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file gllm_privacy_binary-0.4.31-cp313-cp313-manylinux_2_31_x86_64.whl.

File metadata

File hashes

Hashes for gllm_privacy_binary-0.4.31-cp313-cp313-manylinux_2_31_x86_64.whl
Algorithm Hash digest
SHA256 12763e822e3840b14d39c61a7c46fc1dccb22f6a4a624a9cd1600405aefb4a8d
MD5 408a7d69ba8a95d338c356c717b7fbfa
BLAKE2b-256 75a976e156d5ac839584a95b071ba6841893c2553e5e28527adb1e0056d04283

See more details on using hashes here.

File details

Details for the file gllm_privacy_binary-0.4.31-cp313-cp313-macosx_13_0_arm64.whl.

File metadata

File hashes

Hashes for gllm_privacy_binary-0.4.31-cp313-cp313-macosx_13_0_arm64.whl
Algorithm Hash digest
SHA256 c08fbfa831e8f49e301240bee6540e8828085fa0fb9acb0b9ae599ba45e688fa
MD5 bc805ae0c06b9a0d198c6559ed4bc061
BLAKE2b-256 646d5f2b0a88d1c64fd678960969f446a80cc89eed005b15fe9d46f23fe4f1a6

See more details on using hashes here.

Provenance

The following attestation bundles were made for gllm_privacy_binary-0.4.31-cp313-cp313-macosx_13_0_arm64.whl:

Publisher: build-binary.yml on GDP-ADMIN/gl-sdk

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file gllm_privacy_binary-0.4.31-cp312-cp312-win_amd64.whl.

File metadata

File hashes

Hashes for gllm_privacy_binary-0.4.31-cp312-cp312-win_amd64.whl
Algorithm Hash digest
SHA256 01a92da495c9b82b25fa2e167f76562cdf119ec62702b85425f4019733da2b80
MD5 423199b5da97f57dc4022282cfb06dd1
BLAKE2b-256 19f577833a2252fc83bb3c9daec8556fc2babdc9c2cbf20a2f8bf1cde7d265e1

See more details on using hashes here.

Provenance

The following attestation bundles were made for gllm_privacy_binary-0.4.31-cp312-cp312-win_amd64.whl:

Publisher: build-binary.yml on GDP-ADMIN/gl-sdk

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file gllm_privacy_binary-0.4.31-cp312-cp312-manylinux_2_31_x86_64.whl.

File metadata

File hashes

Hashes for gllm_privacy_binary-0.4.31-cp312-cp312-manylinux_2_31_x86_64.whl
Algorithm Hash digest
SHA256 ed68399f165d5f8e2de61d38bc3dc9be93a29007dad187d0a9c0eeed9a936007
MD5 254a9c07c14adebd0bc14d59d92b227b
BLAKE2b-256 85c3278faea8a331e021c7e545402062d2fc7a22f207c2591a4d8787b017cba6

See more details on using hashes here.

File details

Details for the file gllm_privacy_binary-0.4.31-cp312-cp312-macosx_13_0_arm64.whl.

File metadata

File hashes

Hashes for gllm_privacy_binary-0.4.31-cp312-cp312-macosx_13_0_arm64.whl
Algorithm Hash digest
SHA256 a7bbef83b529874d8ec764cd1f3d5bf7520ab81d535a2765c81c3c5fc2978c73
MD5 6d43baeedb8245b79fc502f87c9e55e5
BLAKE2b-256 5182dc46e808d837e0233bb93b9c7dba0775c554cec4fe76fd6ac2b2f0be3975

See more details on using hashes here.

Provenance

The following attestation bundles were made for gllm_privacy_binary-0.4.31-cp312-cp312-macosx_13_0_arm64.whl:

Publisher: build-binary.yml on GDP-ADMIN/gl-sdk

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file gllm_privacy_binary-0.4.31-cp311-cp311-win_amd64.whl.

File metadata

File hashes

Hashes for gllm_privacy_binary-0.4.31-cp311-cp311-win_amd64.whl
Algorithm Hash digest
SHA256 0f96eb1d74114cc649197a63531e79754a1a83c64b13edfbc2ed068766f60116
MD5 6b7db8b072888a0860d34f2822d66b5d
BLAKE2b-256 04146c7d61edf9a7c065f393c07e749c14a90b2c24992b8e2c9856ae226ba095

See more details on using hashes here.

Provenance

The following attestation bundles were made for gllm_privacy_binary-0.4.31-cp311-cp311-win_amd64.whl:

Publisher: build-binary.yml on GDP-ADMIN/gl-sdk

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file gllm_privacy_binary-0.4.31-cp311-cp311-manylinux_2_31_x86_64.whl.

File metadata

File hashes

Hashes for gllm_privacy_binary-0.4.31-cp311-cp311-manylinux_2_31_x86_64.whl
Algorithm Hash digest
SHA256 cff49af0a192ca1b1e4d0137417a20c3aa00591de205c6335bb48f364736b003
MD5 1184de2a189ba3b5fae0a2a26453b119
BLAKE2b-256 61ca468118576eb9981b7c62a046bbb9f020b9095eebb6b2b82994a497d014c9

See more details on using hashes here.

File details

Details for the file gllm_privacy_binary-0.4.31-cp311-cp311-macosx_13_0_arm64.whl.

File metadata

File hashes

Hashes for gllm_privacy_binary-0.4.31-cp311-cp311-macosx_13_0_arm64.whl
Algorithm Hash digest
SHA256 226e793284bfbbf9c5374eeffc79d422e01249936c06798e68625d711f29175b
MD5 e6ec9073d1355ab359a3f53b89ef7375
BLAKE2b-256 345e98ae59f1180d8a1f45b1f6cb61b1b92cf6436471622a75e323f5e7baa5c4

See more details on using hashes here.

Provenance

The following attestation bundles were made for gllm_privacy_binary-0.4.31-cp311-cp311-macosx_13_0_arm64.whl:

Publisher: build-binary.yml on GDP-ADMIN/gl-sdk

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page