CLM

Semantic Token Encoding for LLMs

Compress transcripts, structured data, and system prompts — 60–95% fewer tokens, no model retraining.

CLM is an open-source semantic compression library. It encodes verbose content into compact structured token sequences that LLMs interpret with equal or better accuracy, at a fraction of the token cost.

Three targets, one encoder:

Installation

pip install clm-core

Install the spaCy model for your language:

python -m spacy download en_core_web_sm   # English
python -m spacy download pt_core_news_sm  # Portuguese
python -m spacy download es_core_news_sm  # Spanish
python -m spacy download fr_core_news_sm  # French

If you want the structured-data encoder as part of the same install, add the extra:

pip install "clm-core[sd_encoder]"

Usage

All three encoders share the same interface. CLM auto-detects the input type.

from clm_core import CLMConfig, CLMEncoder

cfg = CLMConfig(lang="en")
encoder = CLMEncoder(cfg=cfg)

Thread Encoder — Transcripts

result = encoder.encode(input_=transcript, metadata={"channel": "voice"})
print(result.compressed)

[INTERACTION:SUPPORT:CHANNEL=VOICE] [DURATION=6m] [LANG=EN]
[DOMAIN:BILLING] [SERVICE:SUBSCRIPTION]
[CUSTOMER_INTENT:REPORT_DUPLICATE_CHARGE] [CONTEXT:EMAIL_PROVIDED]
[AGENT_ACTIONS:ACCOUNT_VERIFIED→DIAGNOSTIC_PERFORMED→REFUND_INITIATED]
[SYSTEM_ACTIONS:PAYMENT_RETRY_DETECTED]
[RESOLUTION:REFUND_INITIATED] [STATE:PENDING_CUSTOMER]
[COMMITMENT:REFUND_3-5_BUSINESS_DAYS] [ARTIFACT:REFUND_REF=RFD-908712]
[SENTIMENT:NEUTRAL→GRATEFUL]

Parse into a structured dict for downstream use:

data = result.to_dict()
# {"channel": "VOICE", "domain": "BILLING", "customerIntent": "REPORT_DUPLICATE_CHARGE",
#  "state": "PENDING_CUSTOMER", "agentActions": [...], "commitments": [...], ...}

Structured Data Encoder (SDE)

SDE was moved to a standalone sub-library. You can find more about it here

catalog = [{"article_id": "KB-001", "title": "Reset Password", "content": "...", "tags": ["security"]}]
result = encoder.encode(catalog)
print(result.compressed)
# {article_id,title,content,tags}[KB-001,Reset Password,To reset your password...,security]

System Prompt Encoder

System prompts are encoded through the same CLMEncoder interface used for the other components. CLM usually classifies the prompt for you, but it helps to think of them as either task prompts or configuration prompts.

cfg = CLMConfig(lang="en")
encoder = CLMEncoder(cfg=cfg)

task_prompt = """
You are a customer service quality analyst.
Analyze call transcripts for compliance issues and sentiment problems.
Return the result as JSON.
"""

task_result = encoder.encode(task_prompt)
print(task_result.metadata["prompt_mode"])
print(task_result.compressed)

If you need the step-by-step guide, start with docs/sys_prompt/index.md.

Task prompts are usually compressed into a single CL token sequence. Configuration prompts can also be bound after compression:

config_prompt = """
<role>You are a helpful support agent</role>

<custom_rules>
Always greet the customer as {{customer_name}}.
</custom_rules>
"""

result = encoder.encode(config_prompt)
bound_prompt = encoder.bind(result, customer_name="Melissa")

print(result.metadata["prompt_mode"])
print(result.compressed)
print(bound_prompt)

Performance

Based on a dataset test across 5,000+ samples:

Thread Encoder

Structured Data Encoder

System Prompt Encoder

Documentation

Official documentation: https://yanickjair.github.io/cllm

Release

This repository publishes two independent Python packages from one main branch:

Use package-specific tags instead of release branches.

Release clm-core

1. Update clm_core/__version__.py.
2. Commit and push the change to main.
3. Create and push a matching tag:

git tag clm_core-v1.0.9
git push origin clm_core-v1.0.9

Release sd_encoder

1. Update the version in crates/sd_encoder/Cargo.toml.
2. Commit and push the change to main.
3. Create and push a matching tag:

git tag sd_encoder-v0.1.0
git push origin sd_encoder-v0.1.0

Pushes to main publish changed packages to TestPyPI. Release tags publish the matching package to PyPI. Both workflows also support manual dispatch with none, testpypi, or pypi.

Star History

License

MIT — see LICENSE.

Issues · Discussions · Contact