labenv-embedding-cache 0.3.2

Shared embedding cache core for cross-project reuse

labenv_embedding_cache

Shared policy and Python cache-core library for projects that reuse text embedding caches.

Files

• embedding_rulebook.yaml: machine-readable cache policy (path/layout/consistency)
• embedding_registry.yaml: machine-readable model registry shared across projects
• embedding_cache_spec.yaml: machine-readable spec for dataset key / metadata / variant tag
• src/labenv_embedding_cache/: reusable Python library
• CODEX_PROMPT.md: reusable prompt template for Codex sessions

Install library

Preferred distribution is wheel (v0.3.2).

# Internal index (recommended)
pip install "labenv-embedding-cache==0.3.2"

# Optional: explicitly point pip to internal index
# PIP_EXTRA_INDEX_URL="https://<internal-index>/simple" pip install "labenv-embedding-cache==0.3.2"

# GitHub Release wheel fallback
pip install "labenv-embedding-cache @ https://github.com/ryuuua/labenv_embedding_cache/releases/download/v0.3.2/labenv_embedding_cache-0.3.2-py3-none-any.whl"

# Local editable (maintainer workflow)
pip install -e /path/to/labenv_embedding_cache

Rollback-only legacy install (VCS pin):

pip install "git+ssh://[e-mail]/ryuuua/labenv_embedding_cache.git@c0154c06ee6e41852c58ac76d6504f5b38d20168#egg=labenv-embedding-cache"

0.3.2 compatibility note

0.3.2 keeps the 0.2.x read compatibility boundary and adds centralized runtime generation through ensure_embedding_cache(..., compute_runtime=True), compute_request_embeddings, and labenv-embedding-cache generate --compute-runtime. Strict canonical verification, safe migration planning, structured tags, generation profiles, and run records remain supported. Existing imports from labenv_embedding_cache.resolver remain available as a compatibility shim.

0.2.0 compatibility note

0.2.0 is an intentional compatibility boundary for legacy fallback.

• Canonical cache files remain supported.
• Legacy tuple caches (arr_0) remain unsupported.
• Legacy index fallback now expects canonical cache metadata on the candidate cache file.
• In practice this means fallback candidates must carry metadata_json with variant_tag and the usual identity metadata.
• Header-only legacy caches that only store ids / embeddings / shuffle_seed plus sparse top-level fields are no longer eligible fallback targets.

If a project still depends on header-only legacy caches, stay on 0.1.x or regenerate those caches into canonical format before upgrading.

Release publishing (PyPI / TestPyPI / GitHub Packages)

Tag push (v*) or manual dispatch triggers:

• .github/workflows/release-wheel.yml (GitHub Release + optional internal index)
• .github/workflows/publish-package-indexes.yml (TestPyPI / PyPI / GitHub Packages)

Before tagging a release:

python tools/check_tag_version.py v0.3.2
PYTHONPATH=src python -m pytest -q
python -m build
python -m twine check dist/labenv_embedding_cache-0.3.2*

Secrets for publish-package-indexes.yml:

• TEST_PYPI_API_TOKEN (optional; if unset, TestPyPI publish is skipped)
• PYPI_API_TOKEN (optional; if unset, PyPI publish is skipped)
• GITHUB_PACKAGES_TOKEN (optional; if unset, GitHub Packages publish is skipped)
• GITHUB_PACKAGES_USERNAME (optional; defaults to ${{ github.actor }})

Standalone run (smoke tests)

Install with optional embedding/debug extras:

pip install -e ".[embed,debug]"

Embedding generation only (no cache):

python tools/embedding_smoketest.py

Embedding + cache read/write:

python tools/embedding_cache_smoketest.py --backend dummy
# or (downloads model)
python tools/embedding_cache_smoketest.py --backend sentence-transformers --model sentence-transformers/all-MiniLM-L6-v2

Debugpy (wait for attach):

python tools/embedding_cache_smoketest.py --backend dummy --debugpy --wait-for-client

Embedding generation from conf/embedding presets (auto DDP/pipeline policy via embedding_model.md):

python tools/generate_embeddings_from_conf.py --preset conf/embedding/qwen3_embedding.yaml --strategy auto

Default is non-normalized embeddings. Use --normalize to generate L2-normalized caches. normalize_embeddings=true is treated as a separate model variant (registry_key=...__l2) and cache variant (norm=l2).

Best practices: docs/EMBEDDING_BEST_PRACTICES.md

Policy path setup

No environment variable is required for normal use. The package resolves its bundled embedding_rulebook.yaml automatically.

If you want to pin EMBEDDING_RULEBOOK_PATH explicitly in your shell profile:

export EMBEDDING_RULEBOOK_PATH="$(labenv-embedding-cache-path rulebook)"

Then reload shell:

source ~/.zshrc

Cache verification, canonical generation, and lock export

Build a read-only index for existing lm/** cache files:

labenv-embedding-cache index-build --cache-dir /work/$USER/data/embedding_cache

Show index stats:

labenv-embedding-cache index-stats --cache-dir /work/$USER/data/embedding_cache

Verify whether request manifests can be served without regeneration:

labenv-embedding-cache verify-requests --requests /path/to/request_manifest.jsonl --min-selected-models 2

Build request manifest rows from canonical metadata in Python:

import labenv_embedding_cache as lec

record = lec.build_request_manifest_entry(
    dataset_name="ag_news",
    model_id="bert",
    model_name="bert-base-uncased",
    expected_cache_path="/work/$USER/data/embedding_cache/lm/bert-base-uncased/ag_news__x.npz",
    metadata=metadata,
)

Export a lock payload for CI/DVC (policy digest + index + optional verify report):

labenv-embedding-cache lock-export \
  --cache-dir /work/$USER/data/embedding_cache \
  --requests /path/to/request_manifest.jsonl \
  --output /work/$USER/data/embedding_cache/.labenv/lock.json \
  --min-selected-models 2

Index file location:

• ${EMBEDDING_CACHE_DIR}/.labenv/index_v1.jsonl

Legacy fallback policy is controlled by rulebook:

• compatibility.legacy_index.enabled
• compatibility.legacy_index.sunset_date
• compatibility.legacy_index.require_ids_sha256_match

The core resolver keeps compatibility.legacy_index.enabled: true for the current compatibility window. New generation workflow commands are canonical-only by default and use fallback only for explicit diagnostics such as fallback-audit or migration validation.

Legacy tuple caches remain outside the reusable canonical contract:

• cache.compatibility.accept_legacy_npz_tuple: false

Identity expansion is controlled by spec:

• identity.profiles.default.hard_fields
• identity.profiles.default.soft_fields
• identity.profiles.default.defaults
• identity.profiles.default.legacy_match_policy

Quick usage

from labenv_embedding_cache.api import get_or_compute_embeddings

vectors, resolution = get_or_compute_embeddings(
    cfg,
    texts,
    ids,
    labels=labels,
    compute_embeddings=my_backend_compute_fn,
)
print(resolution.cache_path, resolution.was_cache_hit)

Generation control profiles

Generation host profiles are defined in generation_profiles.yaml and the packaged policy copy. These profiles do not add public execution targets to this repo.

• env1_docker_embedding / env2_docker_embedding / env3_docker_embedding / env4_docker_embedding: Docker routes with canonical cache root /home/${USER}/data/embedding_cache.
• cc21_slurm_embedding: planner-first Slurm route with cache root /work/${USER}/data/embedding_cache.

For cc21, H100 and H200 are modeled as gpu_family resource attributes. Large GPU jobs prefer H200 and fall back to H100 through planner runtime state when H200 is temporarily unavailable.

V2 migration audit

Existing cache roots should be migrated by auditing and materializing self-describing v2 artifacts, not by guessing identity from legacy filenames. The read-only audit bundle writes only under <cache-root>/.labenv/migration_audits/<run_id>/:

PYTHONPATH=src labenv-embedding-cache migrate-v2-audit \
  --cache-dir /path/to/embedding_cache \
  --run-id readonly-audit-YYYYMMDD

The audit summary reports v2_valid, v1_metadata_upgradeable, metadata_gap_needs_manifest, manual_or_regenerate, and blocked_collision. Non-dry materialization is a later phase and should use only audit-safe copy / write_v2 rows followed by strict validation and explicit legacy quarantine.

Live cache materialization, retirement, and quarantine are intentionally separate gates. Use read-only audits first, preserve the generated audit artifacts, and require explicit rollback evidence before any non-dry retire action.

AppConfig compatibility

• labenv_embedding_cache accepts both Hydra DictConfig and typed dataclass configs (for example CEBRA-NLP-gen2 / larm AppConfig).
• To keep cache sharing stable during typed-migration, dataset fingerprinting treats these legacy-equivalent cases as identical:
  • missing vs empty container: dataset.splits, dataset.label_columns, dataset.label_map, dataset.label_remap
  • missing vs false: dataset.drop_multi_label_samples, dataset.multi_label, dataset.trust_remote_code

Downstream compatibility

CEBRA / Arc / Larm consumers should use the package facade:

import labenv_embedding_cache as lec

The supported public surface and read-only manifest/cache contract are documented in docs/DOWNSTREAM_COMPATIBILITY.md.

Docker / Docker Compose (env1-env4)

Compose files match the labenv_config/envkit-templates profiles (env1_a6000/env2_3090/env3_cc21_a100/env4_cc21_cpu):

• env1: nvcr.io/nvidia/pytorch:25.09-py3 (CUDA 13.0 profile; CUDA 12.8 runtime is frozen legacy)
• env2: nvcr.io/nvidia/pytorch:25.09-py3
• env3: nvcr.io/nvidia/pytorch:23.10-py3
• env4: python:3.11-slim

Cache roots are unified by compose profile:

• env1 / env2: embedding cache = /home/${USER}/data/embedding_cache (EMBEDDING_CACHE_DIR / TEXT_EMBEDDING_CACHE_DIR)
• env3 / env4: embedding cache = /home/${USER}/data/embedding_cache (EMBEDDING_CACHE_DIR / TEXT_EMBEDDING_CACHE_DIR)
• env1 / env2: HF cache reuse = /data/cache (HF_HOME, TRANSFORMERS_CACHE)

Host paths are bind-mounted so caches are reused across repos by default.

Examples:

docker compose -f docker/compose.env1.yaml run --rm app python tools/embedding_cache_smoketest.py --backend dummy
docker compose -f docker/compose.env4.yaml run --rm app python tools/embedding_smoketest.py --backend transformers
# debugpy (port publish requires --service-ports)
docker compose -f docker/compose.env1.yaml run --rm --service-ports app python tools/embedding_cache_smoketest.py --backend dummy --debugpy --wait-for-client

Sweep utility

Create a stable, line-based sweep list file:

python tools/make_sweep_list.py --glob "configs/sweep/*.yaml" --out sweep.txt --root .

Per-project Codex usage

In each project, paste CODEX_PROMPT.md (or add equivalent guidance in AGENTS.md) so Codex always aligns with this policy. For cross-project rollout requests, use PROJECT_MIGRATION_PROMPT.md.

Automation runbook

• Runtime-aware automation prompt: docs/AUTOMATION_PROMPT_RUNTIME_AWARE.md
• Copy/paste execution checklist: docs/AUTOMATION_EXECUTION_CHECKLIST.md
• Pre-filled ready-to-run checklist: docs/AUTOMATION_EXECUTION_CHECKLIST_READY.md
• What to place in central/downstream/runtime repos: docs/REPO_AUTOMATION_ASSETS.md

Architecture governance

• Current structure and dependency DAG: docs/architecture/current-structure.md
• Machine-readable module map: docs/architecture/module-map.yaml
• ML/cache contracts: docs/ML_CONTRACTS.md
• Public/internal API boundary: docs/architecture/public-api.md

Local checks:

python tools/check_policy_bundle_sync.py
python tools/check_architecture_boundaries.py
python tools/architecture_inventory.py
python tools/dynamic_reference_inventory.py --text

Updating shared standards

1. Edit embedding_rulebook.yaml, embedding_registry.yaml, and/or embedding_cache_spec.yaml.
2. In each project, run its embedding-cache validation/tests.
3. If schema behavior changes, bump identity.version in embedding_rulebook.yaml.