GlyphCache

Deterministic exact and Hyper-Glyph-inspired semantic caching for LLM responses.

Package name: glyphcache
Project name: GlyphCache
Install: pip install glyphcache
Import: from glyphcache import PromptCache

GlyphCache is a lightweight, offline-first LLM response cache combining deterministic exact matching with an opt-in Hyper-Glyph-inspired semantic signature index.

It requires no embedding API, vector database, external cache server, or model download.

GlyphCache combines:

• Authoritative exact matching over the complete normalized LLM request.
• Provider-neutral requests with messages, tools, schemas, model parameters, tenant, namespace, and application cache version.
• Deterministic Hash-HDC signatures derived directly from cryptographic hash output rather than model embeddings.
• Immutable semantic prototypes with compact sparse XOR residual encoding.
• LSH candidate filtering that keeps full similarity comparisons bounded.
• Conservative semantic guards for numbers, money, dates, URLs, email addresses, UUIDs, paths, quoted text, and identifiers.
• Memory and SQLite backends with TTL and LRU/size eviction.
• Synchronous and asynchronous APIs with in-process single-flight handling.
• SQLite generation leases for cross-process duplicate suppression.
• Cacheability and redaction hooks for application-specific safety.
• Statistics and event callbacks without raw prompt content by default.
• A small CLI for initialization, inspection, maintenance, health checks, and benchmarks.
• A typed Python API designed for extension through backend, serializer, and encoder protocols.

Before / After

Without GlyphCache, every request reaches the model:

response = call_model(request)

With GlyphCache, compatible cached responses bypass the model call:

from glyphcache import Message, PromptCache, PromptRequest, SQLiteBackend

cache = PromptCache(SQLiteBackend("glyphcache.db"))

request = PromptRequest(
    provider="my-provider",
    model="my-model",
    messages=(
        Message(role="system", content="Answer clearly and concisely."),
        Message(role="user", content="Explain hyperdimensional computing."),
    ),
    params={"temperature": 0, "max_tokens": 500},
)

response = cache.get_or_set(
    request,
    producer=lambda: call_model(request),
    ttl_seconds=86_400,
)

The core job is simple:

Complete request
  -> deterministic exact key
  -> compatible cached response

Exact miss and semantic mode enabled
  -> strict semantic scope
  -> compact deterministic signature
  -> bounded candidates
  -> safety guards and acceptance policy
  -> optional semantic response

The original prompt is never reconstructed from the semantic representation. The exact request hash remains separate and authoritative.

Why Not Just an Embedding Cache?

Embedding caches typically require an embedding API or local model, a vector index, and operational choices around model versions and distance thresholds. GlyphCache uses a different representation: normalized prompt features are bundled into a deterministic bit-packed signature, grouped around immutable prototypes, and filtered through explicit safety controls.

That makes GlyphCache useful when you want a small, inspectable, offline cache layer. It does not claim to understand language better than neural embeddings, and semantic reuse is never treated as proof that two requests are equivalent.

Architecture

                     +----------------------+
                     |    PromptRequest     |
                     +----------+-----------+
                                |
                     +----------v-----------+
                     | Canonical normalizer |
                     +------+---------+-----+
                            |         |
                   exact key|         |semantic scope
                            |         |
                 +----------v--+   +--v----------------+
                 | Exact lookup |   | Hash-HDC encoder |
                 +-------+------+   +--------+---------+
                         |                   |
                      hit|             bit signature
                         |                   |
                         |          +--------v---------+
                         |          | Prototype codec  |
                         |          +--------+---------+
                         |                   |
                         |          +--------v---------+
                         |          | LSH candidates   |
                         |          +--------+---------+
                         |                   |
                         |          +--------v---------+
                         |          | Safety guards    |
                         |          +--------+---------+
                         |                   |
                         +---------+---------+
                                   |
                            Cached response

The exact key includes provider, model, messages, tool definitions, structured-output schema, generation parameters, namespace, tenant, and application cache version. Semantic comparisons remain inside a stricter scope that also fixes system/developer prompts, encoder version, and safety-policy version.

Install

pip install glyphcache

For optional Zstandard compression support:

pip install "glyphcache[compression]"

For documentation dependencies:

pip install "glyphcache[docs]"

For development:

pip install -e ".[dev,compression,docs]"
pytest
python -m build

Quick Start

Exact Cache

from glyphcache import MemoryBackend, Message, PromptCache, PromptRequest

cache = PromptCache(MemoryBackend())
request = PromptRequest(
    provider="example",
    model="example-model",
    messages=(Message("user", "What is HDC?"),),
    params={"temperature": 0},
)

hit = cache.get(request)
if hit is None:
    response = call_model(request)
    cache.set(request, response)
else:
    response = hit.value

Exact matching is enabled by default and always checked first.

Semantic Cache

from glyphcache import CacheConfig, SemanticConfig

config = CacheConfig(
    semantic=SemanticConfig(
        enabled=True,
        similarity_threshold=0.94,
        ambiguity_margin=0.025,
        require_anchor_match=True,
    )
)

cache = PromptCache(MemoryBackend(), config)

Semantic caching is opt-in. By default, it only considers single-user-turn, low-temperature requests without tools or tool messages.

Async Cache

response = await cache.aget_or_set(
    request,
    producer=lambda: call_model_async(request),
    ttl_seconds=86_400,
)

Multi-Tenant Isolation

request = PromptRequest(
    provider="example",
    model="example-model",
    messages=(Message("user", "Summarize my account."),),
    namespace="support",
    tenant_id="customer-42",
)

Tenant and namespace participate in exact keys and semantic scopes, so entries cannot cross those boundaries.

Cacheability Hooks

cache = PromptCache(
    MemoryBackend(),
    should_cache_request=lambda request: request.namespace != "sensitive",
    should_cache_response=lambda request, response: not contains_secret(response),
    redact_metadata=lambda request, metadata: {"trace_id": metadata.get("trace_id")},
)

GlyphCache cannot automatically recognize every secret, permission boundary, one-time code, medical or legal response, or side effect. Applications must use these hooks where appropriate.

CLI

Initialize a SQLite cache:

glyphcache init ./glyphcache.db

Inspect cache totals:

glyphcache stats ./glyphcache.db

Inspect an entry with the response redacted:

glyphcache inspect ./glyphcache.db --entry ENTRY_ID

Explicitly show stored response bytes:

glyphcache inspect ./glyphcache.db --entry ENTRY_ID --show-response

Purge expired entries or a tenant namespace:

glyphcache purge ./glyphcache.db --expired
glyphcache purge ./glyphcache.db --namespace support --tenant customer-42

Run maintenance and health checks:

glyphcache optimize ./glyphcache.db
glyphcache vacuum ./glyphcache.db
glyphcache doctor ./glyphcache.db

Run a local storage benchmark:

glyphcache benchmark --entries 100000 --database ./benchmark.db

Main Features

1. Deterministic Exact Keys

Canonicalization normalizes NFKC Unicode, CRLF/CR line endings, insignificant trailing line whitespace, and mapping insertion order. It rejects NaN and infinity. Output-affecting fields still change the key.

from glyphcache.canonical import exact_cache_key

key = exact_cache_key(request)

2. Hash-HDC Signatures

The default semantic encoder derives bipolar feature vectors directly from SHAKE-256 output. It is role-aware, message-position-aware, token-block-aware, deterministic across processes, and requires no downloaded model.

A default 4,096-bit signature occupies 512 raw bytes before prototype-residual encoding.

Hash-HDC is positioned as deterministic lexical and near-duplicate matching, not general semantic understanding. For genuine semantic input, install glyphcache[embeddings] and use EmbeddingHDCEncoder, or pass a custom PromptEncoder.

3. Prototype-Residual Encoding

Each semantic scope keeps a bounded set of immutable signatures as prototypes. When a signature is close to a prototype, GlyphCache stores the differing bit indices as delta-varints. Dense residuals fall back to the raw signature.

signature:  101101001110...
prototype:  101100001110...
xor:        000001000000...
residual:   [5]

4. Conservative Safety

A semantic hit must satisfy:

• The same strict semantic scope.
• The configured anchor policy.
• A non-expired and non-rejected entry.
• The minimum similarity threshold.
• The ambiguity margin over the second-best candidate.

Hard negatives such as Refund order 123 versus Refund order 124, Python versions, CVE identifiers, invoice IDs, and currency amounts are expected to miss under the default guard policy.

5. SQLite Persistence

The standard-library SQLite backend enables WAL mode, foreign keys, a bounded busy timeout, short transactions, LRU/size eviction, integrity checks, prototype storage, LSH rows, and expiring generation leases.

SQLite is intended for local disk, not arbitrary shared network filesystems.

6. Single-Flight Protection

Concurrent misses for the same exact key share one producer call inside a process. SQLite-backed synchronous producers also use expiring cross-process leases and never hold a write transaction while the model call runs.

7. Statistics and Events

stats = cache.statistics()

print(stats.exact_hits)
print(stats.semantic_hits)
print(stats.misses)
print(stats.stored_bytes)

Events contain IDs, timing, match type, and rejection reason rather than raw prompt content by default.

8. Prepared and Bulk APIs

prepared = cache.prepare(request)
hit = cache.get_prepared(prepared)
cache.set_prepared(prepared, response)

cache.set_many(
    [
        (request_a, response_a),
        (request_b, response_b),
    ]
)

Prepared requests remove repeated canonicalization and hashing from unchanged hot paths. Memory values remain native Python objects; SQLite values are serialized persistently and access accounting is flushed in batches.

Configuration

from glyphcache import CacheConfig, SemanticConfig

config = CacheConfig(
    default_ttl_seconds=86_400,
    max_entries=100_000,
    max_bytes=1_073_741_824,
    store_request_body=False,
    compress_values=True,
    busy_timeout_ms=5_000,
    lease_seconds=120,
    semantic=SemanticConfig(
        enabled=False,
        dimension=4096,
        token_block_size=16,
        max_features=4096,
        similarity_threshold=0.94,
        ambiguity_margin=0.025,
        prototype_threshold=0.78,
        max_candidates=128,
        max_prototypes_per_scope=32,
        lsh_bands=8,
        lsh_bits_per_band=16,
        require_anchor_match=True,
        single_turn_only=True,
        allow_tools=False,
        maximum_temperature=0.2,
        seed=42,
    ),
)

Key settings:

• default_ttl_seconds controls expiry when a write does not supply a TTL.
• max_entries and max_bytes bound backend storage.
• similarity_threshold is the minimum accepted semantic similarity.
• ambiguity_margin rejects a best match that is too close to the runner-up.
• prototype_threshold controls assignment to an existing prototype.
• max_candidates provides a hard bound on full similarity comparisons.
• require_anchor_match protects numeric and identifier-sensitive prompts.
• single_turn_only keeps multi-turn semantic reuse disabled by default.
• allow_tools keeps tool-bearing requests excluded unless explicitly enabled.
• seed makes signature and LSH generation deterministic.

The default threshold is a starting point, not a universal constant. Measure precision and false-hit rate on representative application data.

Benchmarking

The repository includes:

python benchmarks/benchmark_exact.py --entries 10000
python benchmarks/benchmark_semantic.py
python benchmarks/benchmark_storage.py
python benchmarks/benchmark_competitors.py --entries 1000 --probes 5000 --runs 5

The original 1.0.0 comparison and the improved 1.0.1 report are stored in:

• benchmarks/results/competitor_benchmark.md
• benchmarks/results/competitor_benchmark_1.0.1.md
• benchmarks/results/competitor_benchmark_1.0.2.md
• benchmarks/results/semantic_quality.md

The competitor benchmark defaults to five runs and reports medians, sample variance, bulk insertion throughput, SQLite stage timings, and tail operations over one millisecond. For reusable semantic presets, use SemanticConfig.profile("strict"), "balanced", or "recall". Prepared semantic requests cache their encoding with cache.prepare(request, semantic=True).

Semantic evaluation should report precision, recall, F1, false-hit rate, candidate counts, signature time, lookup percentiles, prototype compression rate, and raw-signature fallback rate. Performance reports should identify the hardware, Python version, operating system, concurrency, database warmth, and dataset.

The test suite currently covers exact caching, TTL, semantic retrieval, hard negative guards, signature and codec behavior, SQLite WAL/integrity, eviction, hooks, CLI operations, and sync/async single-flight handling.

Project Structure

src/glyphcache/
  __init__.py             # Stable public API
  __about__.py            # Runtime package version
  cache.py                # PromptCache lookup, write, and single-flight flow
  canonical.py            # Canonical requests, exact keys, semantic scopes
  cli.py                  # Command-line interface
  config.py               # CacheConfig and SemanticConfig
  decorators.py           # Function caching decorator
  exceptions.py           # Package exceptions
  guards.py               # Numeric and identifier anchor extraction
  models.py               # Typed public and internal data models
  policies.py             # Semantic eligibility policy
  statistics.py           # Runtime statistics collector
  backends/
    base.py               # Backend protocol
    memory.py             # In-process backend
    sqlite.py             # SQLite backend and schema
  codec/
    prototypes.py         # Raw and prototype-XOR signature codec
    signature.py          # Hamming similarity
    varint.py             # Varint and delta-index encoding
  encoders/
    base.py               # Encoder protocol
    hash_hdc.py           # Deterministic Hash-HDC encoder
  indexes/
    lsh.py                # Deterministic LSH bands
  serializers/
    base.py               # Serializer protocol
    json.py               # JSON, text, and bytes serializer
  py.typed                # Typing marker
tests/
  test_*.py               # Unit, safety, storage, and concurrency tests
docs/
  index.md                # Documentation home
  concepts.md             # Exact and semantic concepts
  exact-cache.md          # Exact key behavior
  semantic-cache.md       # Semantic pipeline
  safety.md               # Risks and safety policy
  configuration.md        # Configuration reference
  backends.md             # Backend notes
  cli.md                  # CLI reference
  benchmarks.md           # Benchmark methodology
examples/
  basic_cache.py          # Exact in-memory example
  async_cache.py          # Async single-flight example
  semantic_cache.py       # Semantic cache example
  multitenant_cache.py    # Tenant isolation example
  custom_serializer.py    # Serializer extension example
benchmarks/
  benchmark_exact.py      # Exact-cache throughput and latency
  benchmark_semantic.py   # Paraphrase and hard-negative evaluation
  benchmark_storage.py    # SQLite byte accounting
  benchmark_competitors.py # LangChain, GPTCache, and DiskCache comparison
glyphcache.png            # Project logo
pyproject.toml            # Package metadata and dependencies
CHANGELOG.md              # Release history
CONTRIBUTING.md           # Contribution guide
RELEASE.md                # Release checklist
LICENSE                   # MIT license

Development

# Install development, compression, and documentation extras
pip install -e ".[dev,compression,docs]"

# Run tests with the release coverage threshold
pytest --cov=glyphcache --cov-report=term-missing --cov-fail-under=90

# Run linting and formatting checks
ruff check .
ruff format --check .

# Type-check package code
mypy src/glyphcache

# Build and validate distributions
python -m build
twine check dist/*

Security and Limitations

GlyphCache stores response payloads in plaintext unless the application places its SQLite database on encrypted storage. It does not implement encryption, distributed consistency, a hosted cache server, GPU indexing, automatic provider monkey-patching, cross-model reuse, or automatic tool-execution caching.

Semantic caching can return an incorrect response when application-specific differences are not captured by scope or anchors. Keep it disabled for risky request classes and favor false misses over false hits.

License

MIT

Contributing

Contributions are welcome. Open an issue or pull request with the request shape, backend, configuration, expected cache behavior, hard-negative examples, and benchmark or correctness evidence used to evaluate the change.

Citation

If you use GlyphCache in research, please cite:

@software{GlyphCache2026,
  title={GlyphCache: Deterministic Exact and Hyper-Glyph-Inspired Semantic Caching for LLM Responses},
  author={Robert McMenemy},
  year={2026},
  version={1.0.2},
  url={https://github.com/Arkay92/GlyphCache},
}