✂️ Hashformers

Fast, local, multilingual hashtag and identifier segmentation using Transformer language models and beam search.

• +37.5 percentage-point accuracy advantage over Qwen3-0.6B on a fixed 280-item multilingual benchmark
• 14.1 hashtags/second on a single NVIDIA T4
• Available as a Python library, spaCy component, MCP server, and Agent Skill
• Introduced in the original Hashformers paper and recognized as state of the art at LREC 2022

On this page: Quick start · MCP and Agent Skill · When to use Hashformers · Research and citations · Contributing · Resources

Try it: Python Colab ↗ · Codex + Claude Code MCP Colab ↗

Results and recognition: Qwen benchmark · Original paper ↗ · LREC 2022 recognition ↗

Hashformers uses language models and a beam search algorithm to segment text without spaces into words. It fills a gap in the NLP ecosystem between heuristic-based splitters and LLM prompt-based segmentation, and it can use language models from the Hugging Face Model Hub.

🚀 Quick Start

Installation

pip install hashformers

Hashformers requires Python 3.10 or newer and supports Transformers 4.46.1 through 5.x.

Basic Usage

from hashformers import TransformerWordSegmenter as WordSegmenter

ws = WordSegmenter(
    segmenter_model_name_or_path="distilgpt2"
) # You can use any model from the Hugging Face Model Hub

segmentations = ws.segment([
    "#weneedanationalpark",
    "#icecold"
])

print(segmentations)
# ['we need a national park', 'ice cold']

For bulk CUDA workloads, opt into adaptive scorer microbatching independently for beam search and reranking:

ws = WordSegmenter(
    segmenter_model_name_or_path="distilgpt2",
    segmenter_gpu_batch_size="auto",
    segmenter_max_gpu_batch_size=512,
    reranker_model_name_or_path="bert-base-uncased",
    reranker_gpu_batch_size="auto",
    reranker_max_gpu_batch_size=512,
)

Optional Reranking and Fusion

hashtags = ["#icecold"]

# Segmenter only
segmenter_only = ws.segment(
    hashtags,
    use_reranker=False,
)

# Segmenter and reranker with top2 fusion
top2 = ws.segment(
    hashtags,
    fusion_method="top2",
)

# Segmenter and reranker with reciprocal rank fusion
rrf = ws.segment(
    hashtags,
    fusion_method="rrf",
    rrf_k=60,
    fusion_weights={
        "segmenter": 1.0,
        "reranker": 1.0,
    },
)

MCP and Agent Skill

Install the MCP Server

Install and start the optional local MCP server:

pip install "hashformers[mcp]"
hashformers-mcp \
  --model distilgpt2 \
  --batch-size auto \
  --file-root /path/to/project

Connect an MCP Client

Add the server to Codex or Claude Code:

codex mcp add hashformers -- hashformers-mcp --model distilgpt2
claude mcp add --transport stdio --scope user hashformers -- \
  hashformers-mcp --model distilgpt2

Segment Hashtags Interactively

Ask the agent directly:

Use Hashformers to segment #weneedanationalpark and #icecold. Return up to three candidates for each hashtag.

To request default RRF through MCP, configure the server with --reranker-model and pass:

{
  "hashtags": ["#weneedanationalpark", "#icecold"],
  "ranking_strategy": "ensemble",
  "fusion_method": "rrf"
}

Custom rank damping and weights use the same contract for segment_hashtags, start_hashtag_file_job, and rank_candidates:

{
  "ranking_strategy": "ensemble",
  "fusion_method": "rrf",
  "rrf_k": 0,
  "fusion_weights": {"segmenter": 1.0, "reranker": 2.0}
}

Process Large Files

For a large text, CSV, or JSON Lines file, authorize its directory when adding the server:

codex mcp add hashformers -- hashformers-mcp \
  --model distilgpt2 \
  --file-root /path/to/project

Then ask the agent to run the resumable workflow:

Use Hashformers to segment the hashtags in /path/to/project/hashtags.csv. Save the results to /path/to/project/segmented.jsonl and continue until the job is complete.

Select a Model for an Unknown Language

If the language is unknown, let the agent sample the file and select a public Hugging Face model before segmentation:

codex mcp add hashformers -- hashformers-mcp \
  --defer-model-selection \
  --file-root /path/to/project

Sample /path/to/project/hashtags.csv, identify its language, select a compatible public Hugging Face model, and segment the file with Hashformers.

Install the Agent Skill

The repository includes a segment-hashtags Agent Skill. Install it globally for Codex or Claude Code with:

mkdir -p ~/.agents/skills ~/.claude/skills
cp -R .agents/skills/segment-hashtags ~/.agents/skills/
cp -R .agents/skills/segment-hashtags ~/.claude/skills/

Run hashformers-mcp --help for all model, reranker, device, and file-access options.

Using Language-Specific Models

# Russian hashtags with RuGPT3
ws = WordSegmenter(
    segmenter_model_name_or_path="ai-forever/rugpt3small_based_on_gpt2"
)

segmentations = ws.segment(["#москвасити"])

print(segmentations)
# ['москва сити']

spaCy Integration

Hashformers can be used as a spaCy pipeline component:

import spacy
import hashformers.spacy  # registers the "hashformers" component

nlp = spacy.blank("en")
nlp.add_pipe("hashformers", config={"model": "distilgpt2"})

doc = nlp("#weneedanationalpark")
print(doc._.segmented)  # "we need a national park"

Install with spaCy support:

pip install hashformers[spacy]

When to Use Hashformers?

Hashformers occupies the middle ground between CPU heuristics and hosted LLM APIs: it provides model-backed segmentation while keeping inference local and scalable on consumer GPUs.

Hashformers is a strong fit when you have access to GPU compute and work in a niche domain where SymSpell, Ekphrasis, WordNinja, or Spiral (Ronin) is not accurate enough. The cost projections show that even a rented GPU can become competitive with major LLM providers at moderate batch sizes.

For simple domains, a CPU heuristic may be the better choice. For low-volume jobs or maximum accuracy regardless of cost and privacy, a cutting-edge hosted LLM may be a better fit.

📚 Research & Citations

Hashformers was recognized as state-of-the-art for hashtag segmentation at LREC 2022.

Papers Using Hashformers

• Zero-shot hashtag segmentation for multilingual sentiment analysis

• HashSet -- A Dataset For Hashtag Segmentation (LREC 2022)

• Generalizability of Abusive Language Detection Models on Homogeneous German Datasets

• The problem of varying annotations to identify abusive language in social media content

• NUSS: An R package for mixed N-grams and unigram sequence segmentation

• Hate Speech Detection in Turkish and Arabic Languages: A Comprehensive Study

Citation

If you find Hashformers useful, please consider citing our paper:

@misc{rodrigues2021zeroshot,
      title={Zero-shot hashtag segmentation for multilingual sentiment analysis}, 
      author={Ruan Chaves Rodrigues and Marcelo Akira Inuzuka and Juliana Resplande Sant'Anna Gomes and Acquila Santos Rocha and Iacer Calixto and Hugo Alexandre Dantas do Nascimento},
      year={2021},
      eprint={2112.03213},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

🤝 Contributing

Pull requests are welcome! Read our paper for details on the framework architecture.

git clone https://github.com/ruanchaves/hashformers.git
cd hashformers
pip install -e .

📖 Resources

• Qwen and Hashformers Benchmark (August 2026)
• Hosted LLM API Cost Projections (August 2026)
• Benchmark Results and Reproducibility Artifacts (August 2026)
• 15 Datasets for Word Segmentation on the Hugging Face Hub
• Benchmark Scripts
• Evaluation Report (January 2026)
• Evaluation Report (February 2022)