vectorguard-pyhsm 1.1.0

Production-grade software Key Management Service (KMS) — key lifecycle, authenticated encryption, digital signing, and tamper-evident audit logging.

PyHSM

A production-grade software Key Management Service (KMS) providing cryptographic key lifecycle management, authenticated encryption, digital signing, and tamper-evident audit logging.

Available as a Python CLI and library and a production-hardened TypeScript/Node.js library.

---

Why PyHSM

Most applications that need key management face a difficult choice: implement it themselves (error-prone), pay for cloud KMS (vendor lock-in, data sovereignty concerns), or buy a hardware HSM ($20K+, complex). PyHSM is a third path — a well-engineered software KMS that you own, deploy anywhere, and extend freely.

What makes it production-grade:

• AES-256-GCM-SIV encryption (nonce-misuse resistant, TypeScript) / AES-256-GCM with hybrid nonce + AAD binding (Python)
• Argon2id key derivation (OWASP recommended, 64 MB memory-hard)
• HKDF key separation — independent encryption, MAC, and KEK subkeys derived from master
• AES-KWP (RFC 5649) per-key wrapping — keys are double-encrypted at rest in both layers
• Salt-bound KEK derivation — KEK uses a dedicated salt stored inside the encrypted envelope, derived through full PBKDF2 → HKDF path
• Encrypt-then-MAC keystore with HMAC-SHA256 tamper detection
• Pluggable storage backends — file, memory, or custom (database, cloud, etc.)
• Atomic file writes — keystore never corrupts on crash
• Key versioning — rotate without breaking old ciphertexts
• Per-key policies: expiry, operation limits, caller ACLs, rate limiting
• Per-caller ACL enforcement — allowed_callers policy with audit trail on denial
• Per-key concurrency — sharded locks allow parallel operations on different keys
• AAD-bound ciphertext — cryptographically binds ciphertext to key ID and version
• Hybrid nonce strategy — random + counter eliminates birthday-bound collisions
• Input size validation — rejects payloads over 64 MB on both encrypt and decrypt paths
• HMAC-chained append-only audit log with HMAC key derived from master password
• Caller ID tracking — every operation records the caller identity in the audit log
• Deterministic memory zeroization via SecureBytes / SecureBuffer (key material stored as mutable bytearray, not immutable strings)
• Process isolation via Unix domain socket IPC
• Shamir M-of-N master password unlock ceremony
• Startup Known-Answer Tests (KATs) before accepting any operations
• Prometheus metrics
• Backward-compatible ciphertext format versioning (v1 legacy, v2 AAD-bound)
• JWK (RFC 7517) key import/export for interoperability (supports P-256, P-384, P-521, RSA, AES)
• EC P-256, P-384, and P-521 signing with NIST-recommended hash algorithms (SHA-256, SHA-384, SHA-512)
• Fully typed Python API (PEP 561 py.typed marker included)
• 165 tests across both layers

---

Table of Contents

• Python Layer
  • Installation
  • CLI Usage
  • Library Usage
  • Storage Backends
  • Architecture
• TypeScript Layer
  • Installation
  • Library Usage
  • Process Isolation Mode
  • Architecture
• Shared: Shamir Secret Sharing
• Security Model
• Running Tests
• Operations Guide

---

Python Layer

Python Installation

# Install from PyPI
pip install vectorguard-pyhsm

# Or install from source (with pyproject.toml)
pip install .

# For development (includes pytest)
pip install ".[dev]"

CLI Usage

All commands require --store (keystore path) and a master password. The password can be passed via -p or entered interactively at a prompt (recommended for production).

# Generate keys
python cli.py --store keystore.enc generate my-aes-key --type aes-256
python cli.py --store keystore.enc generate my-rsa-key --type rsa-2048
python cli.py --store keystore.enc generate my-ec-key  --type ec-p256
python cli.py --store keystore.enc generate my-ec384   --type ec-p384
python cli.py --store keystore.enc generate my-ec521   --type ec-p521

# Generate a key with a policy
python cli.py --store keystore.enc generate limited-key \
  --type aes-256 \
  --max-operations 500 \
  --expires-at 2027-01-01T00:00:00Z \
  --no-decrypt   # encrypt-only key

# List keys (shows type, current version, creation date)
python cli.py --store keystore.enc list

# Encrypt / Decrypt
python cli.py --store keystore.enc encrypt my-aes-key -d "secret message"
python cli.py --store keystore.enc decrypt my-aes-key -d <ciphertext-hex>

# Pipe via stdin
echo "secret message" | python cli.py --store keystore.enc encrypt my-aes-key

# Sign / Verify (uses stored public key for verify — private key never exposed)
python cli.py --store keystore.enc sign   my-ec-key -d "message to sign"
python cli.py --store keystore.enc verify my-ec-key "message to sign" <sig-hex>

# Export public key (PEM)
python cli.py --store keystore.enc pubkey my-rsa-key

# Rotate an AES key (archives current version, generates new one)
python cli.py --store keystore.enc rotate my-aes-key

# Destroy a key (zeroizes all versions, removes from store)
python cli.py --store keystore.enc delete my-aes-key

# Metrics
python cli.py --store keystore.enc metrics
python cli.py --store keystore.enc metrics --prometheus

# Audit log
python cli.py --store keystore.enc audit                       # dump all entries
python cli.py --store keystore.enc audit --verify              # verify HMAC chain
python cli.py --store keystore.enc audit --operation encrypt   # filter by operation
python cli.py --store keystore.enc audit --key-id my-aes-key   # filter by key
python cli.py --store keystore.enc audit --since 2025-01-01T00:00:00Z

Python Library Usage

from hsm import PyHSM

# Master password is always required — there is no insecure default
hsm = PyHSM(
    storage_path="keystore.enc",
    master_password="your-master-password",
    session_timeout_s=300,      # auto-lock after 5 min inactivity (0 = disabled)
    rate_limit_max_ops=100,     # max ops per key per window
    rate_limit_window_s=60,
)

# Generate keys
hsm.generate_key("aes-key")                          # AES-256 by default
hsm.generate_key("rsa-key", "rsa-2048")
hsm.generate_key("ec-key",  "ec-p256")
hsm.generate_key("ec384",   "ec-p384")              # NIST P-384 (SHA-384)
hsm.generate_key("ec521",   "ec-p521")              # NIST P-521 (SHA-512)

# Generate a key with a policy (including caller ACL)
hsm.generate_key("restricted", policy={
    "allow_encrypt": True,
    "allow_decrypt": False,     # encrypt-only
    "max_operations": 1000,
    "expires_at": "2027-01-01T00:00:00Z",
    "allowed_callers": ["service-a", "service-b"],  # caller ACL
})

# Encrypt / Decrypt (AES-256-GCM with AAD binding and hybrid nonce)
ciphertext = hsm.encrypt("aes-key", "secret message")  # returns hex string
plaintext  = hsm.decrypt("aes-key", ciphertext)        # returns bytes

# All operations support caller_id for audit tracking and ACL enforcement
ciphertext = hsm.encrypt("aes-key", "data", caller_id="my-service")
plaintext  = hsm.decrypt("aes-key", ciphertext, caller_id="my-service")

# Rotate a key (old ciphertexts remain decryptable via version prefix)
new_version = hsm.rotate_key("aes-key")

# Sign / Verify
signature = hsm.sign("ec-key", "message")
is_valid   = hsm.verify("ec-key", "message", signature)  # uses stored public key only

# Sign with P-384 (uses SHA-384 automatically) or P-521 (uses SHA-512)
sig384 = hsm.sign("ec384", "message", caller_id="signer-service")
is_valid = hsm.verify("ec384", "message", sig384, caller_id="verifier")

# Export public key (PEM)
pub_pem = hsm.get_public_key("rsa-key")

# Expiry enforcement (archives expired keys)
hsm.enforce_expiry()

# Metrics
metrics_dict = hsm.get_metrics()
prometheus   = hsm.get_prometheus_metrics()

# Audit log
audit = hsm.get_audit_log()
audit.verify()                                          # returns -1 if clean
entries = audit.export_jsonl(operation="encrypt")       # SIEM-ready list of dicts

# JWK export (RFC 7517) — interoperate with other KMS systems
jwk = hsm.export_jwk("aes-key")                        # {"kty": "oct", "k": "...", ...}
ec_jwk = hsm.export_jwk("ec-key")                      # {"kty": "EC", "crv": "P-256", ...}
ec384_jwk = hsm.export_jwk("ec384")                    # {"kty": "EC", "crv": "P-384", ...}
ec521_jwk = hsm.export_jwk("ec521")                    # {"kty": "EC", "crv": "P-521", ...}

# JWK import — bring keys from external systems
hsm.import_key_jwk("imported-key", {
    "kty": "oct",
    "k": "base64url-encoded-key-material",
    "alg": "A256GCM",
})

# Explicit close (zeroizes master password and key material from memory)
hsm.close_session()

Storage Backends

PyHSM supports pluggable storage backends. The default is file-based with atomic writes, but you can implement custom backends for database, cloud storage, or any other persistence layer.

from hsm import PyHSM, KeyStore
from hsm.backends import StorageBackend, FileBackend, MemoryBackend

# Default: file backend (backward-compatible)
hsm = PyHSM(storage_path="keystore.enc", master_password="pw")

# Explicit file backend
from hsm.backends import FileBackend
store = KeyStore(master_password="pw", backend=FileBackend("/secure/keystore.enc"))

# In-memory backend (for testing or ephemeral use)
from hsm.backends import MemoryBackend
store = KeyStore(master_password="pw", backend=MemoryBackend())

# Custom backend — implement the StorageBackend interface:
#   exists() -> bool
#   read() -> bytes
#   write(data: bytes) -> None
#   delete() -> None

The StorageBackend interface deals only with raw encrypted bytes — all encryption, HMAC verification, and key management logic stays in KeyStore. Backends never see plaintext key material.

Python Architecture

hsm/
  core.py           — PyHSM class: key lifecycle, encrypt/decrypt, sign/verify,
                      per-key AES-KWP wrapping, AAD binding, hybrid nonce,
                      per-key sharded locks, caller_id ACL enforcement
  storage.py        — KeyStore: HKDF key separation (enc/mac/kek subkeys),
                      AES-256-GCM + HMAC-SHA256, cached KEK, pluggable StorageBackend,
                      bytearray key_data for deterministic zeroization
  backends.py       — StorageBackend ABC, FileBackend (atomic writes), MemoryBackend
  secure_memory.py  — SecureBytes: deterministic bytearray zeroization, context manager
  jwk.py            — JWK (RFC 7517) import/export: oct, EC (P-256/P-384/P-521), RSA
  shamir.py         — Shamir secret sharing over GF(256)
  audit.py          — HMAC-chained append-only audit log (HMAC key derived from master password)
  rate_limiter.py   — Sliding-window per-key rate limiter
  metrics.py        — Prometheus-format metrics collector
  self_test.py      — Startup Known-Answer Tests (KATs)
  __init__.py       — Public API exports
  py.typed          — PEP 561 marker for type checker support
cli.py              — Full-featured command-line interface
tests/
  test_pyhsm.py     — 84 pytest tests

---

TypeScript Layer

A production-hardened Node.js library in ./pyhsm-ts/ with additional features: Argon2id KDF, AES-256-GCM-SIV (nonce-misuse resistant), SecureBuffer zeroization, and process isolation mode.

TypeScript Installation

cd pyhsm-ts
npm install
npm run build

Requires Node.js ≥ 18. All dependency versions are pinned exactly.

TypeScript Library Usage

Synchronous constructor (PBKDF2 fallback)

import { PyHSM } from "./pyhsm-ts";

const hsm = new PyHSM({
  storePath: "./keystore.enc",
  masterPassword: "your-master-password",
  sessionTimeoutMs: 300_000,
  backupDir: "./backups",
});

Async factory — Argon2id KDF (recommended for production)

const hsm = await PyHSM.create({
  storePath: "./keystore.enc",
  masterPassword: "your-master-password",
});

The create() factory guarantees Argon2id (64 MB / 3 passes / 4 threads) is used for all key derivation — including the first save on a new keystore. The synchronous constructor falls back to PBKDF2-SHA256 at 480,000 iterations.

Custom storage backend

import { PyHSM, MemoryBackend } from "./pyhsm-ts";

// In-memory backend for testing
const hsm = new PyHSM({
  storePath: "test",
  masterPassword: "pw",
  backend: new MemoryBackend(),
});

// Custom backend — implement the StorageBackend interface:
//   exists(): boolean
//   read(): Buffer
//   write(data: Buffer): void
//   delete(): void

Key operations

// Generate
hsm.generateKey("my-key");

// Generate with policy
hsm.generateKey("restricted", {
  allowEncrypt: true,
  allowDecrypt: true,
  maxOperations: 1000,
  expiresAt: "2027-01-01T00:00:00Z",
  allowedCallers: ["service-a", "service-b"],
});

// Encrypt / Decrypt (AES-256-GCM-SIV, base64 output with version prefix)
const ct = hsm.encrypt("my-key", "secret message");
const pt = hsm.decrypt("my-key", ct);              // returns string

// Rotate (old ciphertexts remain decryptable)
hsm.rotateKey("my-key");

// Destroy (zeroizes all versions)
hsm.destroyKey("my-key");

// Backup and verify
const backupPath = hsm.createBackup();
hsm.verifyBackup(backupPath);   // HMAC + decrypt check without loading into live store

// Metrics
const metrics = hsm.getMetrics();
const prom    = hsm.getPrometheusMetrics();

// Audit log
const audit = hsm.getAuditLog();
const clean  = audit.verify();                     // -1 = no tampering
const events = audit.exportJsonl({ operation: "encrypt", since: "2025-01-01T00:00:00Z" });
const ndjson = audit.toNdjson({ onlyFailed: true }); // SIEM-ready NDJSON string

// Close (zeroizes all Buffers holding sensitive material)
hsm.closeSession();

JWK Import / Export (RFC 7517)

// Export a key as standard JWK — interoperate with any system
const jwk = hsm.exportJwk("my-key");  // {"kty": "oct", "k": "...", "alg": "A256GCM"}

// Import a key from external JWK
hsm.importKeyJwk("external-key", {
  kty: "oct",
  k: "base64url-encoded-key-material",
  alg: "A256GCM",
});

// Import an EC key from another identity provider
hsm.importKeyJwk("idp-signing-key", {
  kty: "EC",
  crv: "P-256",
  x: "...",
  y: "...",
  d: "...",
});

Key ID rules

Key IDs must match ^[a-zA-Z0-9][a-zA-Z0-9._-]{0,127}$:

• 1–128 characters
• Must start with a letter or digit
• May contain letters, digits, ., _, -
• Rejects path traversal (../), prototype pollution (__proto__), spaces

Process Isolation Mode

For maximum security, run the HSM in a separate process. A vulnerability in your application cannot directly read key material in the HSM process's memory.

Start the HSM process:

export PYHSM_MASTER_PASSWORD="your-master-password"
export PYHSM_KEYSTORE_PATH="/secure/keystore.enc"
export PYHSM_SOCKET_PATH="/run/pyhsm/pyhsm.sock"
export PYHSM_CALLER_SECRET="shared-hmac-secret"
export PYHSM_BACKUP_DIR="/secure/backups"

npx tsx pyhsm-ts/process.ts

Connect from your application:

import { PyHSMClient } from "./pyhsm-ts";

const client = new PyHSMClient("/run/pyhsm/pyhsm.sock", "my-service");

await client.generateKey("app-key");
const ct = await client.encrypt("app-key", "secret");
const pt = await client.decrypt("app-key", ct);

await client.rotateKey("app-key");
const path = await client.backup();
const ok   = await client.verifyBackup(path);
const h    = await client.health();
const m    = await client.metrics();

TypeScript Environment Variables

Variable	Default	Description
PYHSM_MASTER_PASSWORD	—	Master password (required unless using PYHSM_SHARES)
PYHSM_SHARES	—	Comma-separated Shamir share JSON objects
PYHSM_KEYSTORE_PATH	./pyhsm-keystore.enc	Encrypted keystore location
PYHSM_AUDIT_LOG_PATH	<storePath>.audit.jsonl	HMAC-chained audit log path
PYHSM_AUDIT_HMAC_KEY	(auto-generated)	Hex 32-byte audit HMAC key
PYHSM_AUDIT_WEBHOOK	—	URL for non-blocking audit event POST
PYHSM_BACKUP_DIR	—	Directory for encrypted backups
PYHSM_SOCKET_PATH	/tmp/pyhsm.sock	Unix domain socket path (IPC mode)
PYHSM_CALLER_SECRET	—	Shared secret for IPC caller HMAC auth
PYHSM_SESSION_TIMEOUT_MS	300000	Idle ms before auto-lock
PYHSM_RATE_LIMIT	100	Max operations per key per window
PYHSM_RATE_WINDOW_MS	60000	Rate limit window duration (ms)
PYHSM_KEY_ID	pyhsm-master	Default key ID for singleton helpers

TypeScript Architecture

pyhsm-ts/
  core.ts             — PyHSM class: key lifecycle, encrypt/decrypt, backup,
                        AES-KWP per-key wrapping, HKDF key separation, pluggable StorageBackend
  storage-backend.ts  — StorageBackend interface, FileBackend, MemoryBackend
  types.ts            — TypeScript interfaces, key ID validation, config with backend option
  jwk.ts              — JWK (RFC 7517) import/export: oct, EC, RSA key types
  shamir.ts           — Shamir secret sharing over GF(256)
  audit.ts            — HMAC-chained audit log, SIEM export
  rate-limiter.ts     — Sliding-window per-key rate limiter
  metrics.ts          — Prometheus metrics collector
  self-test.ts        — Startup Known-Answer Tests (KATs), FIPS mode
  secure-buffer.ts    — SecureBuffer: deterministic Buffer zeroization
  process.ts          — IPC server (process isolation via Unix socket)
  client.ts           — IPC client with HMAC caller auth
  index.ts            — Public API exports and singleton factory
  pyhsm.test.ts       — 81 tests (vitest)
  OPERATIONS.md       — Full operator guide (env vars, deployment, procedures)
  package.json        — Pinned exact dependency versions
  tsconfig.json       — Strict TypeScript configuration

---

Shamir's Secret Sharing

Both layers implement Shamir secret sharing over GF(256) with the AES irreducible polynomial. This can be used to split a master password or any secret into N shares where K are required to reconstruct — and K-1 or fewer shares reveal zero information (information-theoretic security).

Python:

# Split a hex secret into 5 shares, 3 required
python cli.py split -k 3 -n 5 -s "deadbeefcafe..."

# Reconstruct from any 3
python cli.py reconstruct \
  --share '{"index":1,"data":"..."}' \
  --share '{"index":3,"data":"..."}' \
  --share '{"index":5,"data":"..."}'

TypeScript:

import { splitMasterPassword, PyHSM } from "./pyhsm-ts";

// One-time: split the master password into 5 shares, 3 required to unlock
const shares = splitMasterPassword("my-master-password", 3, 5);
// Distribute shares[0..4] to five separate key custodians

// At startup: collect K shares from operators
const hsm = new PyHSM({
  storePath: "./keystore.enc",
  shares: [
    JSON.stringify(shares[0]),
    JSON.stringify(shares[2]),
    JSON.stringify(shares[4]),
  ],
});

Intermediate share buffers are zeroized from memory after reconstruction in both layers.

---

Security Model

Property	Mechanism
Keys encrypted at rest	AES-256-GCM + AAD binding (Python) / AES-256-GCM-SIV (TypeScript)
Per-key double encryption	AES-KWP RFC 5649 wrapping in both layers — keys encrypted inside the encrypted envelope
Keystore tamper detection	Encrypt-then-MAC with separated keys (HKDF-derived enc + mac subkeys)
Key derivation	PBKDF2-SHA256 480k iter → HKDF-Expand (Python) / Argon2id 64MB → HKDF-Expand (TypeScript async)
Key separation	HKDF-Expand with distinct info strings (pyhsm-enc-v1, pyhsm-mac-v1, pyhsm-kek-v1) — encryption, MAC, and KEK keys are cryptographically independent
KEK derivation	Dedicated salt stored inside encrypted keystore → PBKDF2 → HKDF-Expand. KEK is cached in memory for session lifetime and zeroized on close
Memory zeroization	Key material stored as mutable bytearray (Python) / Buffer (TypeScript) with deterministic in-place zeroing. Immutable hex strings eliminated from memory path
Nonce safety	Hybrid nonce: random(4) + counter(4) + random(4) eliminates birthday-bound (Python); AES-256-GCM-SIV nonce-misuse resistant (TypeScript)
Ciphertext binding	AAD ties ciphertext to key_id + version — prevents cross-key confusion attacks
Ciphertext versioning	Format byte distinguishes v2 (AAD-bound) from v1 (legacy) for backward compatibility
Input validation	64 MB maximum enforced on both encrypt (plaintext) and decrypt (ciphertext) paths
Atomic writes	os.replace() (Python) / fs.renameSync on temp file (TypeScript)
Audit integrity	Per-entry HMAC chain; audit HMAC key derived from master password via HKDF (Python) or stored independently (TypeScript)
Caller ID tracking	All operations accept optional caller_id; recorded in every audit entry
Caller ACL enforcement	Per-key allowed_callers policy; unauthorized callers denied with accessDenied audit entry
Constant-time comparisons	hmac.compare_digest (Python) / length-padded timingSafeEqual (TypeScript)
Crypto primitive verification	Known-Answer Tests against RFC vectors at startup
Session isolation	Auto-lock on inactivity; explicit close_session() / closeSession()
Concurrency	Per-key sharded locks (Python) — parallel operations on different keys; serialized save lock prevents write races
Process memory isolation	Optional: IPC mode runs HSM in a separate process (TypeScript)
M-of-N startup ceremony	Shamir split/reconstruct on master password
Pluggable storage	StorageBackend interface — swap file I/O for database, S3, etc.
Key interoperability	JWK (RFC 7517) import/export — supports P-256, P-384, P-521, RSA, AES
EC curve support	P-256 (SHA-256), P-384 (SHA-384), P-521 (SHA-512) — NIST-recommended hash pairing
Type safety	PEP 561 py.typed marker; `str

Honest scope statement: PyHSM is a software KMS. It does not carry FIPS 140-2/3 validation (which requires NIST laboratory certification of the specific binary). It does not provide the physical tamper evidence of a hardware HSM. Key material is protected by OS-level process boundaries, not a secure enclave or physically separate processor. For regulated environments that mandate certified hardware, use a certified HSM; PyHSM is appropriate where software key management is acceptable.

---

Running Tests

Python (pytest):

python -m pytest tests/ -v
# 84 tests

TypeScript (vitest):

cd pyhsm-ts
npm test
# 81 tests

CI runs both suites on every push and pull request, across Python 3.11/3.12/3.13 and Node.js 20. See .github/workflows/ci.yml.

---

Operations Guide

See pyhsm-ts/OPERATIONS.md for the full operator guide, including:

• Deployment architectures (embedded vs. process-isolated)
• All environment variables with descriptions and defaults
• Shamir ceremony procedure
• Key rotation, backup, and backup verification procedures
• Audit log verification and SIEM export
• Prometheus metrics reference
• Security considerations

---

License

MIT