🚀 TurboBatch for Transformers

⚡ کتابخانه‌ای برای تسریع ۱۰ برابری inference مدل‌های transformer با batching هوشمند

⚡ 10x Faster Transformer Inference with Intelligent Dynamic Batching

🇮🇷 فارسی | 🇺🇸 English | 📖 Documentation | 🎯 Examples | 💬 Discussion

🇮🇷 فارسی

قبل از DynamicBatcher: 100 متن → 45 ثانیه ⏰
بعد از DynamicBatcher:  100 متن → 4.5 ثانیه ⚡

pip install turbobatch

git clone https://github.com/Shayanthn/turbobatch.git
cd turbobatch
pip install -e .

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from turbobatch import TurboBatcher

# بارگذاری مدل
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")

# ایجاد TurboBatcher
batcher = TurboBatcher(
    model=model,
    tokenizer=tokenizer,
    max_batch_size=32,
    adaptive_batching=True
)

# متن‌های نمونه
texts = [
    "این محصول فوق‌العاده است!",
    "تجربه بدی بود.",
    "کیفیت خوبی دارد و قیمت مناسب.",
    "خیلی راضی هستم از خرید!"
]

# پیش‌بینی سریع
results = batcher.predict(texts)

for text, result in zip(texts, results):
    sentiment = "مثبت" if result.label == 1 else "منفی"
    print(f"متن: {text}")
    print(f"احساس: {sentiment} (اطمینان: {result.score:.2f})")
    print("-" * 50)

class SentimentAPI:
    def __init__(self):
        self.batcher = DynamicBatcher(model, tokenizer, max_batch_size=64)
    
    def analyze(self, texts):
        return self.batcher.predict(texts)

api = SentimentAPI()
results = api.analyze(["متن اول", "متن دوم", "متن سوم"])

import pandas as pd

# خواندن فایل CSV
df = pd.read_csv("reviews.csv")
texts = df['review_text'].tolist()

# پردازش سریع
results = batcher.predict(texts)

# اضافه کردن نتایج به DataFrame
df['sentiment'] = [r.label for r in results]
df['confidence'] = [r.score for r in results]

# آمار عملکرد
stats = batcher.get_performance_stats()
print(f"📊 تعداد کل batch: {stats['total_batches']}")
print(f"🚀 سرعت: {stats['throughput']:.2f} sample/sec")
print(f"💾 نرخ کش: {stats['cache_hit_rate']:.1f}%")

batcher = DynamicBatcher(
    model=model,
    tokenizer=tokenizer,
    max_batch_size=32,          # حداکثر اندازه batch
    timeout_ms=100,             # timeout برای تشکیل batch
    adaptive_batching=True,     # تطبیق خودکار اندازه batch
    performance_monitoring=True, # مانیتورینگ عملکرد
    enable_caching=True,        # فعال‌سازی کش
    device="cuda"               # استفاده از GPU
)

python examples/sentiment_analysis_demo.py

python examples/advanced_benchmarking_demo.py

🇺🇸 English

🔥 Why DynamicBatcher?

Ever struggled with slow transformer inference in your NLP projects? Tired of waiting hours for your model to process thousands of texts?

DynamicBatcher is your solution! 🎯

Before DynamicBatcher: 100 texts → 45 seconds ⏰
After DynamicBatcher:  100 texts → 4.5 seconds ⚡

✨ Incredible Features

� Quick Installation

pip install turbobatch

Or from source:

git clone https://github.com/Shayanthn/turbobatch.git
cd turbobatch
pip install -e .

� Quick Example - Sentiment Analysis

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from DynamicBatcher import DynamicBatcher

# Load model
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")

# Create DynamicBatcher
batcher = DynamicBatcher(
    model=model,
    tokenizer=tokenizer,
    max_batch_size=32,
    adaptive_batching=True
)

# Sample texts
texts = [
    "I absolutely love this product!",
    "This was a terrible experience.",
    "Good quality and reasonable price.",
    "Highly satisfied with my purchase!"
]

# Fast prediction
results = batcher.predict(texts)

for text, result in zip(texts, results):
    sentiment = "Positive" if result.label == 1 else "Negative"
    print(f"Text: {text}")
    print(f"Sentiment: {sentiment} (Confidence: {result.score:.2f})")
    print("-" * 50)

### 📈 Performance Comparison

🎯 Advanced Examples

1️⃣ Sentiment Analysis API

class SentimentAPI:
    def __init__(self):
        self.batcher = DynamicBatcher(model, tokenizer, max_batch_size=64)
    
    def analyze(self, texts):
        return self.batcher.predict(texts)

api = SentimentAPI()
results = api.analyze(["First text", "Second text", "Third text"])

2️⃣ CSV File Processing

import pandas as pd

# Read CSV file
df = pd.read_csv("reviews.csv")
texts = df['review_text'].tolist()

# Fast processing
results = batcher.predict(texts)

# Add results to DataFrame
df['sentiment'] = [r.label for r in results]
df['confidence'] = [r.score for r in results]

3️⃣ Performance Monitoring

# Performance statistics
stats = batcher.get_performance_stats()
print(f"📊 Total batches: {stats['total_batches']}")
print(f"🚀 Throughput: {stats['throughput']:.2f} samples/sec")
print(f"💾 Cache hit rate: {stats['cache_hit_rate']:.1f}%")

🔧 Advanced Configuration

batcher = DynamicBatcher(
    model=model,
    tokenizer=tokenizer,
    max_batch_size=32,          # Maximum batch size
    timeout_ms=100,             # Batch formation timeout
    adaptive_batching=True,     # Auto-adjust batch size
    performance_monitoring=True, # Enable performance monitoring
    enable_caching=True,        # Enable result caching
    device="cuda"               # Use GPU
)

🎮 Interactive Demo

For quick testing:

python examples/sentiment_analysis_demo.py

For comprehensive benchmarking:

python examples/advanced_benchmarking_demo.py

For Jupyter notebook tutorial:

jupyter notebook examples/DynamicBatcher_Tutorial.ipynb

🤝 Contributing

Want to contribute? Amazing!

1. Fork the repo
2. Create your branch: git checkout -b feature/amazing-feature
3. Commit changes: git commit -m 'Add amazing feature'
4. Push to branch: git push origin feature/amazing-feature
5. Open Pull Request

👨‍💻 Author

Shayan Taherkhani

• 🌐 Website: shayantaherkhani.ir
• 💼 LinkedIn: linkedin.com/in/shayantaherkhani78
• 🎓 Academic Email: shayan.taherkhani@studio.unibo.it
• 📧 Personal Email: shayanthn78@gmail.com

⭐ Support the Project

If you found this project helpful:

• ⭐ Star this repo
• 🍴 Fork it
• 📢 Share with friends
• 🐛 Report bugs
• 💡 Suggest features

📊 Project Stats

💰 Commercial Use & Monetization

DynamicBatcher can be monetized through:

• 🏢 Enterprise Consulting: Offer optimization services for large-scale NLP deployments
• ☁️ SaaS Solutions: Build high-performance NLP APIs with faster inference
• 🎓 Training & Workshops: Teach high-performance NLP techniques
• 📊 Custom Solutions: Develop tailored batching strategies for specific use cases
• 💼 Performance Auditing: Optimize existing NLP pipelines for enterprises

🔒 Security & Licensing

• ✅ MIT Licensed: Free for commercial and personal use
• 🔐 No data collection: Your data stays private
• 🛡️ Enterprise ready: Suitable for production environments
• 📝 Well documented: Comprehensive documentation and examples

📜 License

This project is licensed under the MIT License. See LICENSE for details.

🙏 Acknowledgments

• HuggingFace team for the amazing transformers library
• PyTorch team for the incredible framework
• Open source community for inspiration and support

@software{taherkhani2025dynamicbatcher,
  author = {Taherkhani, Shayan},
  title = {DynamicBatcher: High-Performance Dynamic Batching for Transformer Models},
  year = {2025},
  url = {https://github.com/Shayanthn/turbobatch}
}

batcher = DynamicBatcher(tokenizer, collate_fn=custom_collate)

Mixed Precision Support
```bash
batcher = DynamicBatcher(tokenizer, fp16=True)  # Enable AMP

Progress Tracking

batches = batcher.create_batches(texts, progress_bar=True)

🧩 Integration Guide With PyTorch DataLoader

from torch.utils.data import DataLoader

class TextDataset:
    def __init__(self, texts):
        self.texts = texts
    
    def __len__(self):
        return len(self.texts)
    
    def __getitem__(self, idx):
        return self.texts[idx]

dataset = TextDataset(texts)
dataloader = DataLoader(
    dataset,
    batch_sampler=DynamicBatchSampler(dataset, tokenizer, batch_size=32),
    collate_fn=batcher.dynamic_collate
)

With FastAPI Web Service

from fastapi import FastAPI
app = FastAPI()
batcher = DynamicBatcher(tokenizer)

@app.post("/predict")
async def predict(texts: List[str]):
    batches = batcher.create_batches(texts)
    results = []
    for batch in batches:
        outputs = model(**batch[0])
        results.extend(process_outputs(outputs))
    return {"predictions": results}

📚 Documentation DynamicBatcher Class

DynamicBatcher(
    tokenizer: AutoTokenizer,
    max_sequence_length: int = 512,
    fp16: bool = False,
    progress_bar: bool = False,
    sorting_strategy: str = 'ascending'  # or 'descending'
)

🎯 Use Cases:

🔍 Document Processing Pipelines
💬 Real-time Chat Applications
📰 News Article Classification
🗣 Speech-to-Text Post Processing
🌍 Multilingual Translation Services

📬 Contact

Shayan Taherkhani
📧 shayanthn78@gmail.com
💼 LinkedIn
🐙 GitHub