ESMC-LI

Fast, local protein retrieval with ESM-C residue embeddings and late interaction (MaxSim).

ESMC-LI is a pip-installable protein retrieval engine for local and offline search. It encodes proteins with ESM-C 300M, projects every residue into a 128-dimensional vector, and ranks candidates with exact MaxSim late interaction. The index is a single memory-mapped .esmci file; inference runs through esmc.cpp.

Late interaction follows ColBERT; the protein-specific model idea was introduced by ProtoCol (arXiv:2605.29158). ESMC-LI focuses on the practical index, full-length evaluation, and a reproducible local package.

Paper: Scaling Late-Interaction Protein Retrieval · API: docs/api.md · Model: AnanyaPathak/esmc-li-300m-homology-v1 · Indexes: AnanyaPathak/esmc-li-indexes

At a glance

Install

pip install esmc-li

# optional, macOS: fused Metal search
pip install "esmc-li[metal]"

PyPI wheels target macOS arm64 and Linux x86_64 (glibc 2.34+), Python 3.10+. There is no Windows or Intel-macOS wheel in v1. Wheels bundle the native runtime and projection head; no compiler or separate esmc.cpp build is needed.

The fine-tuned GGUF is downloaded from Hugging Face on first use. The current model is ~664 MB and is cached under ~/.cache/esmc-retrieval (the historical cache path is kept for compatibility). Subsequent runs are local.

For a source install, libesmc is not bundled. Install from GitHub and set ESMC_LIB or use the ESMC_EMBED CLI fallback; see CONTRIBUTING.md.

Quickstart

esmc-li index pull scope-1k

esmc-li search scope-1k \
  --sequence "GERRRSQLDRDQCAYCKEKGHWAKDCPKKPRGPRGPRPQT" \
  --top-k 5

from esmc_li import ProteinIndex

index = ProteinIndex.load("scope-1k")
hits = index.search("GERRRSQLDRDQCAYCKEKGHWAKDCPKKPRGPRGPRPQT", k=5)
print(hits[0])

Build an index from your own FASTA:

esmc-li build proteins.fasta --output proteins.esmci
esmc-li search proteins.esmci \
  --sequence "MKTIIALSYIFCLVFADYKDDDDK" \
  --top-k 10

Batch FASTA queries:

esmc-li search proteins.esmci \
  --query queries.fasta \
  --output results.tsv

How it works

1. esmc.cpp encodes a sequence into residue rows [L × 960] with BOS/EOS kept.
2. A frozen linear head projects each row to 128 dimensions and L2-normalizes it.
3. Rows are packed into an .esmci mmap index as f16.
4. Queries use the same encoder and head; exact MaxSim produces the top-k.
5. Metal accelerates the max-and-reduce on macOS; numpy is the portable fallback.

Long chains are chunked at 2046 residues with 128-residue overlap. The index format is documented in docs/esmci.md.

How we compare

• ProtoCol: the released code is vendored at vendor/ProtoCol and run through scripts/repro_protocol.py; only the device is changed.
• MMseqs2: -s 7.5 -e 1e10 --max-seqs 1000.
• MinHash: 5-mers, 256 permutations.
• Pooled ESM-C: mean residue vector, cosine similarity.
• Frozen MaxSim: ESM-C residue vectors with no trained projection.
• ESMC-LI: fine-tuned ESM-C suffix, 128D head, exact MaxSim.

All reported values use the same frozen split; see the paper for the full protocol and limitations.

Results

SCOPe cR@k curves and full-length 100K comparison. Values are from the committed evaluation CSVs; see the paper for protocol details.

SCOPe uses ASTRAL40, n=2314, leave-one-out superfamily retrieval, and a three-seed mean for the fine-tuned row. Full-length uses 100K single-clan Swiss-Prot chains and 500 test-clan queries; its ESMC-LI row is seed 0. See the paper for the full protocol, baselines, and limitations.

A useful negative result: frozen ESM-C is weaker than frozen ESM-2 35M under the same harness (0.6304 vs 0.8215 cR@10). The ESM-C gain appears after fine-tuning; the model is not a drop-in frozen upgrade.

ProtoCol reproduction and frozen-backbone control on our SCOPe split. The ProtoCol rows use the vendored code in vendor/ProtoCol/.

Performance

M4 Max, fused Metal, 50 queries. The 1M row is batch-mode; build and storage, not query latency at 100K, are the practical constraints. A 1K demo index searches in ~19 ms/query.

Exact search, coarse rerank, and index size versus database size on M4 Max.

Platform support

Python 3.10+. Wheels include libesmc and ggml; macOS binaries are unsigned and print a warning.

Model and indexes

The wheel contains no model weights. Model weights and GGUF derivatives are governed by the EvolutionaryScale Cambrian Open License; see NOTICE and CAMBRIAN-OPEN-LICENSE.md. ESMC-LI is not affiliated with EvolutionaryScale.

Python API

from esmc_li import ProteinIndex

# Build once
index = ProteinIndex.from_fasta("proteins.fasta", output="proteins.esmci")

# Load later
index = ProteinIndex.load("proteins.esmci")

hits = index.search("MKTIIALSYIFCLVFADYKDDDDK", k=20)
batch = index.search_batch(["MKT...", "MKT..."], k=20)

search() returns:

[{"id": "protein_123", "score": 0.87, "metadata": {...}}, ...]

See docs/api.md for signatures, device selection, and index compatibility.

CLI

esmc-li build proteins.fasta --output proteins.esmci
esmc-li search proteins.esmci --sequence "MKT..." --top-k 10
esmc-li search proteins.esmci --query queries.fasta --output results.tsv
esmc-li benchmark proteins.esmci
esmc-li index list
esmc-li index pull scope-1k

Limitations

• Exact search is interactive at 100K and batch-oriented at 1M.
• No Windows or Intel-macOS wheel in v1.
• No GPU retrieval; Metal is used for the fused max kernel on macOS.
• The model download is ~664 MB on first use.
• macOS runtime binaries are unsigned.
• The model weights and derivatives are Cambrian-licensed; the Python package itself is MIT.

Reproducibility and development

The paper and its tables are generated from a pinned results lock. The short version:

make audit      # tests + paper claims + citation + lock checks
make tables     # regenerate paper tables and figures

Development setup, the full paper audit, release steps, and benchmark commands are in CONTRIBUTING.md.

Citation

@software{pathak2026esmcli,
  author  = {Pathak, Ananya},
  title   = {ESMC-LI: Fast Local Protein Retrieval with ESM-C Residue Embeddings and Late Interaction},
  version = {1.0.1},
  year    = {2026},
  url     = {https://github.com/AnanyaP-WDW/esmc-li}
}

See CITATION.cff for the machine-readable entry.

License and acknowledgements

Code is MIT-licensed. See LICENSE. ESM-C 300M weights and GGUF derivatives are governed by the Cambrian Open License; a copy is included in CAMBRIAN-OPEN-LICENSE.md.

Built with ESM. Thanks to the esmc.cpp, ESM-C, SCOPe, and UniProt teams.