langformers 0.5.0

🚀 Unified NLP Pipelines for Language Models

Langformers

Langformers is a flexible and user-friendly library that unifies NLP pipelines for both Large Language Models (LLMs) and Masked Language Models (MLMs) into one simple API.

What makes Langformers special? Whether you're generating text, training classifiers, labelling data, embedding sentences, reranking sentences, or building a semantic search index... the API stays consistent:

from langformers import tasks

component = tasks.create_<something>(...)
component.<do_something>()

No need to juggle different frameworks — Langformers brings Hugging Face Transformers, Ollama, FAISS, ChromaDB, Pinecone, and more under one unified interface.

Use the same pattern everywhere:

tasks.create_generator(...)   # Chatting with LLMs
tasks.create_labeller(...)    # Data labelling using LLMs
tasks.create_embedder(...)    # Embeding Sentences
tasks.create_reranker(...)    # Reranking Sentences
tasks.create_classifier(...)  # Training a Text Classifier
tasks.create_tokenizer()      # Training a Custom Tokenizer
tasks.create_mlm(...)         # Pretraining an MLM
tasks.create_searcher(...)    # Vector Database search
tasks.create_mimicker(...)    # Knowledge Distillation
tasks.create_chunker(...)     # Chunking for LLMs

Installation

Langformers can be installed using pip.

pip install -U langformers

This installs the latest version with core dependencies.

Optional Dependencies

Langformers includes optional integrations you can install depending on your use case:

• For FAISS support: pip install -U langformers[faiss]

• For ChromaDB support: pip install -U langformers[chromadb]

• For Pinecone support: pip install -U langformers[pinecone]

• To install all optional features: pip install -U langformers[all]

Supported Tasks

Below are the pre-built NLP tasks available in Langformers. Each link points to an example in Langformer's documentation to help you get started quickly.

Generative LLMs (e.g., Llama, Mistral, DeepSeek)

• Seamless Chat with LLMs
• LLM Inference via API
• Data Labelling with LLMs
• Chunking

Masked Language Models (e.g., RoBERTa)

• Train Text Classifiers
• Pretrain MLMs from scratch
• Continue Pretraining MLMs on custom data

Embeddings & Search (e.g., Sentence Transformers, FAISS, Pinecone)

• Embed Sentences
• Semantic Search
• Rerank Sentences
• Mimic a Pretrained Model (Knowledge Distillation)

Documentation

Complete documentation and advanced usage examples are available at: https://langformers.com.

License

Langformers is released under the Apache License 2.0.

Contributing

We welcome contributions! Please see our contribution guidelines for details.

---

Built with ❤️ for the future of language AI.