devlacruz_hashlib 🛡️

devlacruz_hashlib (v0.2.0) is an enterprise-grade AI Privacy Shield & TOON Compressor, engineered in Rust with PyO3. It serves as a Zero-Trust middleware for Python AI Agents, encrypting sensitive data locally before reaching commercial LLMs (OpenAI, Anthropic, DeepSeek) while slashing prompt token costs with tabular compression.

---

Table of Contents

1. The Problem It Solves
2. What's New in v0.2.0
3. Features
4. Installation
5. Quick Start
6. Streaming LLM Responses (FSM Decoder)
7. Deterministic Encryption & Referential Integrity
8. Real-World AI Agent Integration
9. API Reference
10. Performance
11. Backwards Compatibility
12. Contributing
13. License & Author

---

The Problem It Solves

When developing AI Agents (AutoGen, LangChain, CrewAI, or custom tool-calling agents), sending PII (Personally Identifiable Information) or PHI (Protected Health Information) to external model APIs introduces critical regulatory risks (GDPR, HIPAA, SOC-2). Additionally, sending verbose JSON structures wastes tokens on repetitive keys and punctuation.

devlacruz_hashlib resolves both challenges seamlessly at native Rust speed:

1. Locally Encrypts sensitive fields using ChaCha20-Poly1305 DAEAD.
2. Compresses data into the compact TOON format (or standard JSON if needed).
3. Preserves Referential Integrity: Identical entities produce identical tokens, enabling the LLM to group, sort, and reason consistently.
4. Resilient Streaming: Decrypts responses in real time chunk-by-chunk with zero latency overhead.
5. Anti-BPE Fuzzy Matcher: Automatically heals tokens affected by tokenizer splits, Markdown escapes (\<...\>), spaces, or Unicode zero-width characters.

---

What's New in v0.2.0

• 🧠 DAEAD Deterministic Encryption (SIV-like): Preserves entity equivalence across prompts using HMAC-derived nonces and Authenticated Additional Data (AAD).
• 🏷️ Semantic Typed Tokens: Tokens specify category tags (e.g. <VAULT:NAME:...>, <VAULT:CONDITION:...>), preventing context confusion.
• ⚡ Streaming VaultStreamDecoder: Finite State Machine (FSM) for zero-latency, chunk-by-chunk decryption over WebSockets / Server-Sent Events (SSE).
• 🛡️ BPE & Markdown Resilience: Fuzzy scanner transparently recovers corrupted tokens without failing decryption.
• 🧹 Memory Protection with zeroize: Key material wrapped in SessionKey with automatic zeroization upon cleanup or context exit.
• 🔄 Hybrid Output: Support for both tabular toon and native json format outputs.
• 🔒 100% Backwards Compatible: Retains flawless compatibility with legacy v0.1.x code signatures and string keys.

---

Features

• 🔒 Zero-Trust Local Cryptography: ChaCha20-Poly1305 DAEAD. Master keys reside strictly in local memory and are never transmitted.
• 📉 Token Optimization: Slashes JSON token consumption by up to 60% with TOON compression.
• ⚡ Rust Native Execution: Drops the Python GIL (py.allow_threads) during heavy operations.
• 📦 Universal ABI3 Wheels: Works out of the box on Python 3.8 through 3.14+ across Linux, macOS, and Windows.

---

Installation

pip install devlacruz_hashlib

---

Quick Start

Basic Usage (v0.1.x Compatible)

import devlacruz_hashlib as shield
import json

data = [
    {"id": 1, "name": "Alejandro De La Cruz", "diagnosis": "Anxiety", "score": 85},
    {"id": 2, "name": "Sam Altman", "diagnosis": "Flu", "score": 92}
]

# Encrypt sensitive keys into TOON format
toon_payload, key = shield.prepare_for_ai(json.dumps(data), ["name", "diagnosis"])

print(toon_payload)
# data[2]{diagnosis,id,name,score}
# <VAULT_...>,1,<VAULT_...>,85
# <VAULT_...>,2,<VAULT_...>,92

# Decrypt LLM response
llm_reply = "Patient <VAULT_...> requires follow-up care."
restored = shield.restore_from_ai(llm_reply, key)
print(restored)
# Patient Alejandro De La Cruz requires follow-up care.

---

Streaming LLM Responses (FSM Decoder)

When consuming tokens in real time from OpenAI, Anthropic, or Ollama streams, tokens may arrive sliced across chunk boundaries. Use VaultStreamDecoder:

from devlacruz_hashlib import SessionKey, VaultStreamDecoder, prepare_for_ai
import json

data = [{"id": 1, "patient": "John Doe", "condition": "Migraine"}]
payload, session_key = prepare_for_ai(
    json.dumps(data),
    ["patient"],
    deterministic=True,
    semantic_types={"patient": "NAME"}
)

# Initialize stream decoder
decoder = VaultStreamDecoder(session_key)

# Simulated stream fragments
stream_chunks = [
    "Patient ",
    "<VAULT:NAME:ab",
    "cd12345",
    "...>",
    " reported mild pain."
]

for chunk in stream_chunks:
    output = decoder.decode_chunk(chunk)
    print(output, end="", flush=True)

# Flush any trailing buffer
print(decoder.flush(), end="")
# Output: "Patient John Doe reported mild pain."

---

Deterministic Encryption & Referential Integrity

In agentic loops, LLMs often need to compare entities across rows. Standard randomized nonces turn the same name into two different tokens, causing hallucinations. With deterministic=True:

import devlacruz_hashlib as shield
import json

records = [
    {"record_id": 101, "customer": "Alice Smith"},
    {"record_id": 102, "customer": "Alice Smith"},  # Same identity
    {"record_id": 103, "customer": "Bob Jones"}
]

json_str, key = shield.prepare_for_ai(
    json.dumps(records),
    ["customer"],
    format="json",
    deterministic=True,
    semantic_types={"customer": "NAME"}
)

data = json.loads(json_str)
# data[0]['customer'] == data[1]['customer'] -> True (<VAULT:NAME:same_hash>)

---

Secure Memory with SessionKey

Keys can be explicitly cleared from memory or managed via Python context managers:

from devlacruz_hashlib import SessionKey, prepare_for_ai
import json

with SessionKey() as key:
    payload, _ = prepare_for_ai(
        json.dumps([{"secret": "API_KEY_123"}]),
        ["secret"],
        deterministic=True
    )
    # Perform operations...

# 'key' is automatically zeroized from RAM after exiting the block
assert key.is_zeroized()

---

Real-World AI Agent Integration

import devlacruz_hashlib as shield
import json

INTERNAL_DB = {
    "Alejandro De La Cruz": {"balance": 5000, "status": "VIP"},
    "Sam Altman": {"balance": 1000, "status": "Standard"}
}

def secure_get_customer_data(encrypted_name: str, key: str) -> str:
    # 1. Decrypt locally
    real_name = shield.restore_from_ai(encrypted_name, key).strip()
    if real_name in INTERNAL_DB:
        return f"User {encrypted_name}: {json.dumps(INTERNAL_DB[real_name])}"
    return "User not found."

# Protect user prompt
masked_input, session_key = shield.prepare_for_ai(
    json.dumps([{"target": "Alejandro De La Cruz"}]),
    ["target"],
    deterministic=True
)

# Agent receives token and retrieves data securely
agent_response = "Summary for <VAULT:...>: balance 5000."
final_response = shield.restore_from_ai(agent_response, session_key)

---

API Reference

Functions

Function	Signature	Description
prepare_for_ai	(json_str: str, sensitive_keys: list[str], *, format: str = "toon", deterministic: bool = False, semantic_types: dict[str, str] = None) -> tuple[str, SessionKey | str]	Encrypts sensitive keys and formats output as TOON or JSON. Returns payload and session key. Drops GIL.
restore_from_ai	(text: str, master_key: SessionKey | str) -> str	Scans text for <VAULT...> tokens, applies fuzzy healing, and returns plaintext. Compatible with both legacy and typed tokens. Drops GIL.

Classes

• SessionKey(): Wrapper for ChaCha20 master key with zeroize capabilities and context manager support (__enter__, __exit__).
• VaultStreamDecoder(key: SessionKey | str): FSM decoder for processing fragmented token streams in real time.

---

Performance

Compiled natively in Rust with SIMD and ABI3 optimizations.

Benchmarking 10,000 JSON records (Parsing + Encryption + Compression):

• Pure Python (cryptography + json): ~2.5 - 4.0s
• devlacruz_hashlib (Rust + PyO3): ~0.08 - 0.15s (~25x faster)

---

Backwards Compatibility

devlacruz_hashlib v0.2.0 is fully backwards-compatible with earlier v0.1.x versions:

1. Calls to prepare_for_ai(json_str, keys) default to format="toon" and return tuple[str, str].
2. restore_from_ai seamlessly accepts legacy str keys and <VAULT_...> tokens.
3. Passing raw str keys issues a gentle DeprecationWarning while executing successfully without crashing.

---

Contributing

Contributions are welcome!

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m "feat: enhance streaming decoder")
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

---

License

This project is licensed under the MIT License. See the LICENSE file for details.

---

Author

Alejandro De La Cruz
✉️ devlarood@gmail.com
GitHub: @DevLaCruz