VLM Engine
A high-performance Python package for Vision-Language Model (VLM) based content tagging and analysis. This package provides an advanced implementation for automatic content detection and tagging, delivering superior accuracy compared to traditional image classification methods.
Features
- Remote VLM Integration: Connects to any OpenAI-compatible VLM endpoint (no local model loading required)
- Modest local footprint: Inference runs on your VLM server, not on this machine—no multi‑GB model weights or local GPU VRAM for the vision model. Video is decoded in a streaming, frame-by-frame way (PyAV / decord), so RAM stays bounded by active frames and caches rather than the whole file. Works well on CPU-only and low-spec laptops when paired with a remote endpoint.
- Context-Aware Detection: Leverages Vision-Language Models' understanding of visual relationships for accurate content tagging
- Flexible Architecture: Modular pipeline system with configurable models and processing stages
- Asynchronous Processing: Built on asyncio for efficient video and image processing
- Customizable Tag Sets: Easy configuration of detection categories
- Production Ready: Includes retry logic, error handling, and comprehensive logging
Installation
From Source
git clone https://github.com/Haven-hvn/haven-vlm-engine-package.git
cd haven-vlm-engine-package
pip install -e .
Requirements
- Python 3.10+
- Compatible VLM server endpoint:
- Remote OpenAI-compatible API (recommended)
- Local server using LM Studio
- Haven's custom VLM available at https://havenmodels.orbiter.website/
Quick Start
import asyncio
from vlm_engine import VLMEngine
from vlm_engine.config_models import EngineConfig, ModelConfig
# Configure the engine
config = EngineConfig(
active_ai_models=["llm_vlm_model"],
models={
"llm_vlm_model": ModelConfig(
type="vlm_model",
model_id="HuggingFaceTB/SmolVLM-Instruct",
api_base_url="http://localhost:7045",
tag_list=["tag1", "tag2", "tag3"] # Your custom tags
)
}
)
# Initialize and use
async def main():
engine = VLMEngine(config)
await engine.initialize()
results = await engine.process_video(
"path/to/video.mp4",
frame_interval=2.0,
threshold=0.5
)
print(f"Detected tags: {results}")
asyncio.run(main())
Multiplexer Configuration (Load Balancing)
For high-performance deployments, you can configure multiple VLM endpoints with automatic load balancing:
from vlm_engine.config_models import EngineConfig, ModelConfig
config = EngineConfig(
active_ai_models=["vlm_multiplexer_model"],
models={
"vlm_multiplexer_model": ModelConfig(
type="vlm_model",
model_id="HuggingFaceTB/SmolVLM-Instruct",
use_multiplexer=True, # Enable multiplexer mode
multiplexer_endpoints=[
{
"base_url": "http://server1:7045/v1",
"api_key": "",
"name": "primary-server",
"weight": 5, # Higher weight = more requests
"is_fallback": False
},
{
"base_url": "http://server2:7045/v1",
"api_key": "",
"name": "secondary-server",
"weight": 3,
"is_fallback": False
},
{
"base_url": "http://backup:7045/v1",
"api_key": "",
"name": "backup-server",
"weight": 1,
"is_fallback": True # Used only when primaries fail
}
],
tag_list=["tag1", "tag2", "tag3"]
)
}
)
Architecture
Core Components
-
VLMEngine: Main entry point for the package
- Manages model initialization and pipeline execution
- Handles asynchronous processing of videos and images
-
VLMClient: OpenAI-compatible API client with multiplexer support
- Supports any VLM with chat completions endpoint
- Load balancing across multiple endpoints using multiplexer-llm
- Automatic failover for high availability
- Includes retry logic with exponential backoff and jitter
- Handles image encoding and prompt formatting
-
Pipeline System: Flexible processing pipeline
- Modular design allows custom processing stages
- Built-in support for preprocessing, analysis, and postprocessing
- Configurable through YAML or Python objects
-
Model Management: Dynamic model loading
- Supports multiple model types (VLM, preprocessors, postprocessors)
- Lazy loading for efficient resource usage
- Thread-safe model access
Configuration
Basic Configuration
from vlm_engine.config_models import EngineConfig, ModelConfig, PipelineConfig
config = EngineConfig(
active_ai_models=["my_vlm_model"],
models={
"my_vlm_model": ModelConfig(
type="vlm_model",
model_id="model-name",
api_base_url="http://localhost:8000",
tag_list=["action1", "action2", "action3"],
max_new_tokens=128,
request_timeout=70,
vlm_detected_tag_confidence=0.99
)
},
pipelines={
"video_pipeline": PipelineConfig(
inputs=["video_path", "frame_interval"],
output="results",
models=[{"name": "my_vlm_model", "inputs": ["frame"], "outputs": "tags"}]
)
}
)
Multiplexer Benefits
- Load Balancing: Distribute requests across multiple VLM endpoints based on configurable weights
- High Availability: Automatic failover to backup endpoints when primary endpoints fail
- Improved Performance: Parallel processing across multiple servers for higher throughput
- Seamless Integration: Drop-in replacement for single endpoint configurations
- Flexible Configuration: Mix of primary and fallback endpoints with custom weights
Advanced Configuration
The package supports complex configurations including:
- Multiple models in a pipeline
- Custom preprocessing and postprocessing stages
- Category-specific settings (thresholds, durations, etc.)
- Batch processing configurations
See the examples directory for detailed configuration examples.
For comprehensive multiplexer setup and configuration, see MULTIPLEXER_INTEGRATION.md.
API Reference
VLMEngine
class VLMEngine:
def __init__(self, config: EngineConfig)
async def initialize()
async def process_video(video_path: str, **kwargs) -> Dict[str, Any]
Processing Parameters
video_path: Path to the video fileframe_interval: Seconds between frame samples (default: 0.5)threshold: Confidence threshold for tag detection (default: 0.5)return_timestamps: Include timestamp information (default: True)return_confidence: Include confidence scores (default: True)
Performance Optimization
The VLM runs on the server you configure; locally you mostly pay for decode, resizing, and API I/O. If CUDA is present, frame tensors may use the GPU; otherwise preprocessing uses the CPU.
Memory Requirements
- Video preprocessing does not load the full decoded video into RAM. PyAV (macOS ARM) decodes sequentially; elsewhere decord reads and decodes frames on demand from disk.
- Working set is dominated by currently decoded frames (plus small decoder/caches inside FFmpeg/decord), tensors passed to the VLM, and optional frame caching (e.g. the frame extractor keeps a small bounded cache).
- Very high resolution or many parallel workers can still increase RAM and CPU; for huge files, tune
frame_intervalor concurrency rather than assuming linear growth with file size on disk.
API Optimization
- Configure retry settings based on your VLM server's capacity
- Adjust
max_new_tokensto balance speed vs accuracy - Use appropriate
frame_intervalto reduce processing time and API calls
Processing Speed
- Increase
frame_intervalto sample fewer frames (faster but less accurate) - Use batch processing when your VLM endpoint supports it
- Consider running multiple VLM instances for parallel processing
Extending the Package
Custom Models
Create custom model classes by inheriting from the base Model class:
from vlm_engine.models import Model
class CustomModel(Model):
async def process(self, inputs):
# Your custom processing logic
return results
Custom Pipelines
Define custom pipelines for specific use cases:
custom_pipeline = PipelineConfig(
inputs=["image_path"],
output="analysis",
models=[
{"name": "preprocessor", "inputs": ["image_path"], "outputs": "processed_image"},
{"name": "analyzer", "inputs": ["processed_image"], "outputs": "analysis"}
]
)
Troubleshooting
Common Issues
-
Connection Errors
- Ensure your VLM server is running and accessible
- Check the
api_base_urlconfiguration - Verify firewall settings
-
Slow Processing
- Increase frame interval for faster processing
- Optimize VLM server settings
Logging
Enable debug logging for troubleshooting:
import logging
logging.basicConfig(level=logging.DEBUG)
Contributing
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
Development Setup
git clone https://github.com/yourusername/vlm-engine.git
cd vlm-engine
pip install -e ".[dev]"
Running Tests
pytest tests/
License
This project is licensed under the MIT License - see the LICENSE file for details.
Acknowledgments
-
Built on top of modern Python async patterns
-
Inspired by production ML serving architectures
-
Haven's custom VLM models trained using SmolVLM-Finetune - Model Download found on https://havenmodels.orbiter.website/
-
Designed for integration with OpenAI-compatible VLM endpoints
Support
For issues and feature requests, please use the GitHub issue tracker.
For questions and discussions, join our community:
- Discord: Link to Discord
Note: This package requires an OpenAI-compatible VLM endpoint. Options include:
Remote Services
- Any OpenAI-compatible API endpoint
- Akash deployment - https://github.com/Haven-hvn/haven-inference
Local Setup
- LM Studio - Easy local VLM hosting with OpenAI-compatible API
The package does not load VLM models directly - it communicates with external VLM services via API.
Release files for vlm-engine 1.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| vlm_engine-1.1.0.tar.gz | 73.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| vlm_engine-1.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 155.9 kB
Release files / vlm_engine-1.1.0.tar.gz
| Download URL | vlm_engine-1.1.0.tar.gz |
|---|---|
| Size | 73.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d7982e7084065ae2bd22885efda30a7d5280f0e72b06db750908402c875803bb
|
|
BLAKE2b-256 checksum How to use checksums |
304669e9156cab9fbf6afffbd6bcc5261b028789a7407a57aed3b5c411b3742d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 13, 2026.
Transparency logRelease files / vlm_engine-1.1.0-py3-none-any.whl
| Download URL | vlm_engine-1.1.0-py3-none-any.whl |
|---|---|
| Size | 82.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
9adcf46589de44c9dd6b979c1bf052b2e947cb7517b1181c0659b0a8a3fdebbf
|
|
BLAKE2b-256 checksum How to use checksums |
eb4f20b0537f75e72da4a2fb9e09596c7f760d493beef1fa55194aae06ce21bf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 13, 2026.
Transparency log
