PromptCapsule 📦

Lossless prompt capsules for sharing, retrieval, and agent-to-agent handoff (v0.1.4 — fail-closed integrity, size limits, vault & Gist hardening).

Overview

PromptCapsule is a generic, reusable, open-source tool for compressing and reliably reconstructing LLM prompts. It bridges the gap between short-prompt compression and long-prompt storage, giving developers an honest, transparent way to share, version-control, and manage their AI prompts.

The Problem

When people talk about "prompt compression," they usually mean two fundamentally different things:

Type	What It Does	Trade-off
Type 1: True Compression	Small prompts (≤500 bytes) packed inline with zlib + Base85	No external storage; size reduction is entropy-limited
Type 2: Key-Based Retrieval	Long prompts stored in a vault; agents share a short capsule key	Requires a shared backend; the handle is ~99% smaller than a multi‑KB prompt

PromptCapsule gives you BOTH, transparently and automatically.

The Solution

PromptCapsule uses a hybrid approach:

Short prompts (≤500 bytes UTF-8):
  INPUT PROMPT → ZLIB COMPRESS → BASE85 ENCODE → Portable capsule string
  (self-contained, no storage needed)

Long prompts (>500 bytes):
  INPUT PROMPT → STORE IN VAULT (SQLite, GitHub Gist, S3) → Generate short hash key
  (retrieves from configured backend)

Both modes are automatic — just compress once and the library picks the best strategy. Both are verified — checksums ensure byte-for-byte exact reconstruction.

Agent-to-agent handoff

A capsule is a message payload. One agent packs a prompt; another unpacks it and continues only if verification succeeds.

result = pc.decompress(capsule, vault_backend=vault)  # strict=True by default
# raises IntegrityError if checksum fails — fail closed for agent handoffs

• Inline (cap_i_…): self-contained. No shared store. Best for short instructions (≤500 bytes).
• Vault (cap_v_…): both agents use the same backend (SQLite, GitHub Gist, or S3). The long prompt stays in the vault; only a short key moves between agents.
• Use strict=False only if you intentionally want plaintext with verified=False (legacy).

This is a use of the existing API, not a separate agent protocol, and not encryption. The project includes 81 automated tests (core + security regressions). The number 27+ historically referred to core unit tests — it is test coverage, not capsule length.

---

What's new in 0.1.4 (follow-up hardening)

Addresses remaining items from the 0.1.3 fix-verification review:

ID	Change
F13	Inline Base85 must round-trip; trailing junk (_EXTRA) is rejected
F07	Vault bind failure returns empty text even with strict=False (no cross-agent plaintext leak)
F11	GitHubGistBackend(require_owner=True) by default; optional allowed_gist_ids allowlist

Still open by design: F09 (8-hex prefix is not a MAC — use IntegrityChecker HMAC if you need authenticity). Demo bus F03–F05 remain out of package scope.

---

What's new in 0.1.3 (security & limits documentation)

Version 0.1.2 hardened the library against the issues in our security review. 0.1.3 publishes the same protections with clear PyPI/README documentation of limits, what was fixed, and what is still out of scope.

Size & operational limits

Limit	Value	Behavior
Inline vs vault threshold	500 bytes (UTF-8)	≤500 → inline capsule; >500 → requires a vault backend
Max prompt on compress	10 MiB (MAX_PROMPT_SIZE)	Larger inputs raise ValueError
Max capsule string size	10 MiB	Oversized capsules rejected on decompress
Max zlib expansion	10 MiB (MAX_DECOMPRESSED_SIZE)	Blocks zip/zlib bombs
Checksum in capsule	8 hex chars (SHA-256 prefix)	Integrity signal, not a MAC / not encryption
Vault keys	Unguessable (secrets.token_urlsafe)	Not sequential counters

from promptcapsule import PromptCapsule, IntegrityError

pc = PromptCapsule()
print(pc.INLINE_THRESHOLD)       # 500
print(pc.MAX_PROMPT_SIZE)        # 10485760
print(pc.MAX_DECOMPRESSED_SIZE)  # 10485760

Security issues addressed (library)

Issue	Status	Mitigation
Integrity fail-open (plaintext when checksum fails)	Fixed (0.1.2+)	decompress(strict=True) raises IntegrityError
Empty checksum prefix → verified=True	Fixed	Exactly 8 lowercase hex chars required
zlib bomb / unbounded decompress	Fixed	Cap at 10 MiB expansion
Huge compress DoS	Fixed	MAX_PROMPT_SIZE = 10 MiB
Predictable vault keys	Fixed	secrets.token_urlsafe keys
Vault key swap → other agent’s text	Fixed (0.1.4)	Fail-closed + bind; strict=False returns empty text
Trailing Base85 junk (_EXTRA)	Fixed (0.1.4)	Round-trip encode check
S3 confused-deputy keys	Fixed	Prefix + .. guard
Gist capsule-controlled IDs	Fixed (0.1.4)	require_owner=True + optional allowlist
HMAC helper NameError	Fixed	Module-level hmac

Remaining limitations (read carefully)

PromptCapsule is packaging + retrieval, not a security boundary by itself:

• Not encryption. Capsules and vault contents are readable to anyone who can obtain them or access the vault.
• Not authentication. There is no built-in agent identity, API keys, or mTLS in the library.
• 8-hex checksum is truncated SHA-256, not an HMAC. Use IntegrityChecker HMAC helpers (and your own secrets) if you need authenticity beyond integrity.
• Vault handoffs require a shared backend both agents can reach, with correct ACLs (file perms, private gists, IAM).
• Custom backends must implement retrieve_with_checksum for full key↔checksum binding.
• Demo “capsule bus” HTTP services (if you build one) need their own auth, rate limits, and body size caps — that is outside this package.

Recommended agent pattern

from promptcapsule import PromptCapsule, IntegrityError

pc = PromptCapsule()
try:
    result = pc.decompress(capsule, vault_backend=vault)  # strict=True
except IntegrityError:
    raise  # do not act on untrusted / tampered prompts

# only then use result.text

---

Features

✨ Hybrid Compression

• Short prompts: Inline zlib + Base85 compression
• Long prompts: Vault storage with automatic fallback
• Automatic mode selection based on size

🤝 Agent-to-agent handoff

• Pass a capsule instead of the full prompt
• Receiver reconstructs the original text and checks verified
• Inline capsules travel alone; vault capsules use a shared backend

🔒 Integrity Verification

• SHA256 checksums for all capsules
• Byte-for-byte exact reconstruction guarantee
• Verification status in decompression results

🎯 Honest Positioning

• Transparent about what it actually does (capsule strings, not LLM output compression)
• Clear comparisons vs. existing tools (LLMLingua, LangChain Hub, gzip)
• No false claims of novelty

🌍 Pluggable Backends

• In-Memory: Great for testing and prototyping
• SQLite: Local storage, no external dependencies
• GitHub Gist: Cloud storage, version control friendly
• AWS S3: Enterprise-grade scalability (optional)

⚙️ Python 3.8+

• Pure Python, minimal dependencies
• Cross-platform compatible
• Type hints throughout

---

Installation

# Basic installation (includes in-memory and SQLite backends)
pip install promptcapsule

# With cloud storage support
pip install promptcapsule[vault]

Requirements

• Python 3.8+
• No external dependencies for core functionality
• Optional: boto3 for S3 backend, PyGithub for GitHub Gist backend

---

Quick Start

Basic Usage (Short Prompts)

from promptcapsule import PromptCapsule

pc = PromptCapsule()

# Compress a prompt
prompt = "You are a helpful Python coding assistant."
capsule = pc.compress(prompt)
# capsule: "cap_i_a1b2c3d4_K*i0?5Z7....."

# Decompress anywhere, on any device/account
result = pc.decompress(capsule)
assert result.text == prompt
assert result.verified is True  # Integrity verified ✓

Long Prompts with Vault Backend

from promptcapsule import PromptCapsule
from promptcapsule.backends import SQLiteBackend

pc = PromptCapsule()
vault = SQLiteBackend("prompts.db")

# Compress a long, carefully-crafted prompt
long_prompt = """
You are an expert in machine learning...
[2000+ characters of detailed context]
"""

capsule = pc.compress(long_prompt, vault_backend=vault)
# capsule: "cap_v_a1b2c3d4_sql_20240921_120000_0001"

# Decompress later (data retrieves from SQLite)
result = pc.decompress(capsule, vault_backend=vault)
assert result.text == long_prompt
assert result.verified is True

Real-World Scenario: Version-Controlling Prompts

import json
from promptcapsule import PromptCapsule
from promptcapsule.backends import SQLiteBackend

pc = PromptCapsule()
vault = SQLiteBackend("prompts.db")

# Iterate on your prompts over time
prompts_history = {
    "v1.0": pc.compress(
        "Generate a blog post about AI",
        vault_backend=vault
    ),
    "v1.1": pc.compress(
        "Generate a blog post about AI, focused on practical applications",
        vault_backend=vault
    ),
    "v1.2": pc.compress(
        "Generate a technical blog post about AI/ML, 2000+ words, with code examples",
        vault_backend=vault
    ),
}

# Save to git
with open("prompts.json", "w") as f:
    json.dump(prompts_history, f)

# Later (or different branch), retrieve and decompress
with open("prompts.json", "r") as f:
    history = json.load(f)

for version, capsule in history.items():
    result = pc.decompress(capsule, vault_backend=vault)
    print(f"{version}: {result.text[:50]}...")

---

How It Works

Inline Mode (Short Prompts)

1. Compress: prompt → zlib (level 9) → Base85 encode → capsule
2. Format: cap_i_<8-char checksum>_<encoded data>
3. Verify: Decompress → compare checksum prefix
4. Result: Portable, self-contained capsule string

Example:

Original: "Hello world!" (12 bytes)
Capsule:  "cap_i_2f575c63_EZxgxD/'" (28 bytes)
Ratio:    ~2.3x (expected for very short content)

Vault Mode (Long Prompts)

1. Store: prompt → Save to backend (SQLite/Gist/S3) → Generate key
2. Format: cap_v_<8-char checksum>_<backend key>
3. Retrieve: On decompress → fetch from backend using key
4. Verify: Compare checksum prefix with retrieved content

Example:

Original:  "You are a senior software architect..." (2,000 chars)
Capsule:   "cap_v_a1b2c3d4_sql_20240921_120000_0001" (41 chars)
Ratio:     ~48x compression (key is permanent pointer to vault)

Automatic Mode Selection

pc = PromptCapsule()

# Anything ≤ 500 bytes uses inline
pc.compress("short prompt")  # cap_i_...

# Anything > 500 bytes needs vault
pc.compress("A" * 600)  # Error! Need vault_backend=

# With vault, auto-selects best mode
pc.compress("short prompt", vault_backend=backend)  # Still cap_i_...
pc.compress("A" * 600, vault_backend=backend)       # cap_v_...

---

Backends

In-Memory Backend

Perfect for testing and prototyping:

from promptcapsule.backends import InMemoryBackend

backend = InMemoryBackend()
capsule = pc.compress("long prompt" * 100, vault_backend=backend)

SQLite Backend

Local, file-based storage:

from promptcapsule.backends import SQLiteBackend

backend = SQLiteBackend("prompts.db")
# Auto-creates schema, stores prompts locally
capsule = pc.compress("long prompt" * 100, vault_backend=backend)

GitHub Gist Backend

Cloud storage with version control:

from promptcapsule.backends import GitHubGistBackend

backend = GitHubGistBackend(token="github_pat_...")
# Stores as private gist, returns gist ID
capsule = pc.compress("long prompt" * 100, vault_backend=backend)

Requires: pip install promptcapsule[vault]

AWS S3 Backend

Enterprise-grade cloud storage:

from promptcapsule.backends import S3Backend

backend = S3Backend(bucket="my-prompts", region="us-east-1")
# Stores in S3, returns S3 key
capsule = pc.compress("long prompt" * 100, vault_backend=backend)

Requires: pip install promptcapsule[vault]

---

Command-Line Interface

# Compress a prompt
echo "Your prompt here" | promptcapsule compress
# Output: cap_i_a1b2c3d4_...

# Decompress a capsule
promptcapsule decompress "cap_i_a1b2c3d4_..."
# Output: Your prompt here

# With vault backend
promptcapsule compress --vault sqlite:prompts.db < prompt.txt
promptcapsule decompress --vault sqlite:prompts.db "cap_v_..."

---

Comparison with Existing Tools

Tool	What It Does	Strength	Limitation
LLMLingua	Lossy semantic compression of prompts	High compression ratios	Approximate reconstruction, no exact guarantee
LangChain Hub	Cloud-hosted prompt template sharing	Easy sharing, versioning	Locked into LangChain ecosystem
gzip/zlib directly	General-purpose compression	Simple, standard	Can't store large prompts, no key-based retrieval
PromptCapsule	Hybrid lossless + vault-based retrieval	Exact reconstruction, honest, pluggable	Requires storage backend for long prompts

---

Use Cases

1️⃣ Share Prompts Across Accounts

Move carefully-crafted prompts between work and personal accounts without copy-pasting:

capsule = pc.compress(my_favorite_prompt)
# Share via email, Slack, message, etc.
# Later, paste in personal account
result = pc.decompress(capsule)

2️⃣ Version-Control Your Prompts

Keep prompts in git alongside your code:

prompts/
  ├── article-writer-v1.0.cap
  ├── article-writer-v1.1.cap
  └── article-writer-v2.0.cap

3️⃣ Portable Prompt Library

Share a repo of prompts that works on any machine, any account:

prompts = load_capsule_library("prompts.json")
for name, capsule in prompts.items():
    result = pc.decompress(capsule, vault_backend=my_vault)
    print(f"{name}: {result.text}")

4️⃣ Automated Prompt Management

Build CI/CD pipelines that validate and archive prompts:

# Compress before commit
for prompt_file in glob("*.txt"):
    with open(prompt_file) as f:
        prompt = f.read()
    capsule = pc.compress(prompt, vault_backend=vault)
    save_to_metadata(capsule)

# Retrieve during deployment
capsule = load_from_metadata()
prompt = pc.decompress(capsule, vault_backend=vault).text

---

API Reference

PromptCapsule Class

class PromptCapsule:
    def compress(
        self,
        text: str,
        vault_backend: Optional[VaultBackend] = None,
    ) -> str:
        """
        Compress a prompt into a capsule string.
        
        Args:
            text: The prompt to compress
            vault_backend: Backend for storing long prompts (required if > 500 bytes)
        
        Returns:
            Capsule string (starts with "cap_")
        
        Raises:
            ValueError: If text is empty or > 500 bytes without vault
            TypeError: If text is not a string
        """
    
    def decompress(
        self,
        capsule: str,
        vault_backend: Optional[VaultBackend] = None,
    ) -> CapsuleResult:
        """
        Decompress a capsule back to the original prompt.
        
        Args:
            capsule: The capsule string to decompress
            vault_backend: Backend for retrieving long prompts
        
        Returns:
            CapsuleResult with:
                - text: Original prompt
                - verified: Integrity check passed
                - mode: "inline" or "vault"
                - checksum: SHA256 of original
                - original_size: Bytes
                - capsule_size: Bytes
        
        Raises:
            ValueError: If capsule format is invalid
            KeyError: If vault key not found
        """

CapsuleResult Named Tuple

class CapsuleResult(NamedTuple):
    text: str              # Original prompt
    verified: bool         # Checksum matched
    mode: str              # "inline" or "vault"
    checksum: str          # SHA256 hash
    original_size: int     # Bytes
    capsule_size: int      # Bytes

VaultBackend Abstract Class

Implement to create custom backends:

class VaultBackend:
    def store(self, text: str, checksum: str) -> str:
        """Store text, return a key."""
        raise NotImplementedError
    
    def retrieve(self, key: str) -> str:
        """Retrieve text by key."""
        raise NotImplementedError

---

Testing

Run the full regression test suite:

python run_tests.py

Expected output:

======================================================================
PromptCapsule - Regression Test Suite
======================================================================

Core Functionality Tests:
✓ Compress short prompt (inline mode)
✓ Reject empty strings
✓ Reject invalid types
[... 24 more tests ...]

Test Results: 27/27 passed
======================================================================

---

Architecture Notes

Why This Design?

1. Hybrid approach: Best of both worlds — inline compression for portability, vault for scale
2. Honest positioning: We don't claim to be better than lossy compression at reducing LLM cost — we solve a different problem (portability + exact reconstruction)
3. Pluggable backends: Future-proof; use SQLite today, S3 tomorrow, custom backend next week
4. Integrity by default: Every capsule includes a checksum; verification is automatic
5. Zero external dependencies for core functionality — just zlib and base64, both stdlib

Checksum Strategy

• Uses SHA256 (64 hex chars)
• Stores first 8 chars in capsule for quick verification
• Prevents accidental corruption detection
• Does not provide cryptographic authentication (future: optional HMAC signing)

Compression Levels

• Zlib level 9: Maximum compression
• Base85: Better human readability than Base64 (4-char savings per 80 bytes)
• Trade-off: ~2-3x size increase for very short content (overhead of capsule format)

---

Roadmap

• CLI tool with full feature parity
• HMAC signing for optional authentication
• Async backend support
• Compression format versioning (for future improvements)
• Web UI for managing vaults
• Prompt templates + variable interpolation
• Analytics: track prompt reuse, version adoption

---

Security Considerations

• Integrity: ✅ Checksums detect corruption
• Authenticity: ⚠️ No signing (roadmap)
• Confidentiality: ⚠️ Vault contents transmitted/stored in plaintext (use HTTPS, encrypted S3, private gists)
• Access Control: Depends on backend (GitHub: private gists, S3: IAM policies)

Recommendation: Treat capsule strings like URLs — they're short but semantically empty. Don't rely on them for security-critical operations.

---

License

MIT License - see LICENSE file

---

Contributing

Contributions welcome! Please:

1. Write tests for new features
2. Follow PEP 8 style guide
3. Add docstrings
4. Update README with examples

---

Citation

If you use PromptCapsule in your research or project, please cite:

@software{promptcapsule2024,
  title={PromptCapsule: Open-source prompt compression and retrieval library},
  author={Nirogi, Udaya},
  year={2024},
  url={https://github.com/UdayaNirogi/promptcapsule}
}

---

FAQ

Q: How is this different from just using a URL shortener? A: URL shorteners store data on a third-party server. PromptCapsule lets you choose your own backend (SQLite locally, S3 privately, GitHub Gist for sharing, etc.). Plus, checksums guarantee integrity.

Q: Can I use this to compress LLM outputs? A: No — that's a different problem (lossy compression). PromptCapsule is for inputs (prompts), not outputs.

Q: Is the capsule string secure? A: No — treat it like a URL. The 8-char checksum prefix is for integrity, not authentication. If you need signing, that's a roadmap item.

Q: What about very old Python versions? A: We support Python 3.8+. Older versions should still work (no fancy syntax), but we don't test them.

Q: Can I use multiple backends at once? A: Yes! Just pass different backends to different compress/decompress calls. Each backend is independent.

---

See Also

• LLMLingua — Lossy prompt compression
• LangChain Hub — Prompt management platform
• zlib Documentation — Compression format

---

Feedback & Support

• 📖 Documentation
• 🐛 Report Issues
• 💬 Discussions
• 📧 Email: udaya@example.com

---

Made with ❤️ for developers who care about their prompts.