openvino-easy 1.0.0

Framework-agnostic Python wrapper for OpenVINO 2025

OpenVINO-Easy 🚀

Framework-agnostic Python wrapper for OpenVINO 2025

Load and run AI models with three functions:

import oe

oe.load("runwayml/stable-diffusion-v1-5")   # auto-download & convert
img = oe.infer("a neon cyber-city at night")       # chooses NPU>GPU>CPU  
stats = oe.benchmark()                              # JSON perf report

🎯 Installation

Pick the variant that matches your hardware:

# CPU-only (40MB wheel, fastest install)
pip install "openvino-easy[cpu]"
# or
pip install "openvino-easy[runtime]"

# Intel® Arc/Xe GPU support
pip install "openvino-easy[gpu]"

# Intel® NPU support (Arrow Lake/Lunar Lake with FP16-NF4)
pip install "openvino-easy[npu]"

# With INT8 quantization support
pip install "openvino-easy[quant]"

# Audio model support (Whisper, TTS)
pip install "openvino-easy[audio]"

# Full development environment (OpenVINO, NNCF, optimum ~1GB)
pip install "openvino-easy[full]"

# Everything (for development)
pip install "openvino-easy[all]"

🩺 Installation Troubleshooting

Something not working? Run the doctor:

# Comprehensive diagnostics
oe doctor

# Get fix suggestions for specific hardware
oe doctor --fix gpu
oe doctor --fix npu

# JSON output for CI systems
oe doctor --json

# Check device status
oe devices

Common issues:

Problem	Solution
ImportError: OpenVINO runtime not found	Install with hardware extras: pip install "openvino-easy[cpu]"
NPU detected but not functional	Install Intel NPU drivers from intel.com
GPU detected but not functional	Install Intel GPU drivers (intel-opencl-icd on Linux)
NNCF not available for INT8 quantization	Install quantization support: pip install "openvino-easy[quant]"
FP16-NF4 not supported	Requires Arrow Lake/Lunar Lake NPU with OpenVINO 2025.2+
Version warnings	Upgrade OpenVINO: pip install --upgrade "openvino>=2025.2,<2026.0"
PyTorch model (.pt/.pth) not loading	Convert to ONNX first: torch.onnx.export(model, dummy_input, "model.onnx") then oe.load("model.onnx")
"Native PyTorch model conversion failed"	Upload to Hugging Face Hub with config.json or use ONNX format for best compatibility

📦 What Each Variant Includes

Variant	OpenVINO Package	Size	Best For
[cpu] / [runtime]	openvino runtime	~40MB	Production deployments, CPU-only inference
[gpu]	openvino runtime	~40MB	Intel GPU acceleration
[npu]	openvino runtime	~40MB	Intel NPU acceleration
[quant]	openvino + NNCF	~440MB	INT8 quantization support
[audio]	openvino + librosa	~100MB	Audio models (Whisper, TTS)
[full]	openvino + NNCF + optimum	~1GB	Development, model optimization, research

⚡ Quick Start

Basic Usage

import oe

# Load any model (Hugging Face, ONNX, or OpenVINO IR)
oe.load("microsoft/DialoGPT-medium")

# Run inference (automatic tokenization for text models)
response = oe.infer("Hello, how are you?")
print(response)  # "I'm doing well, thank you for asking!"

# Benchmark performance
stats = oe.benchmark()
print(f"Average latency: {stats['avg_latency_ms']:.2f}ms")
print(f"Throughput: {stats['throughput_fps']:.1f} FPS")

# Explicitly free memory when done
oe.unload()

Advanced Usage

# Specify device preference and precision
oe.load(
    "runwayml/stable-diffusion-v1-5",
    device_preference=["NPU", "GPU", "CPU"],  # Try NPU first, fallback to GPU, then CPU
    dtype="fp16-nf4"  # New FP16-NF4 precision for Arrow Lake/Lunar Lake NPUs
)

# Generate image
image = oe.infer(
    "a serene mountain landscape at sunset",
    num_inference_steps=20,
    guidance_scale=7.5
)

# Get detailed model info
info = oe.get_info()
print(f"Running on: {info['device']}")
print(f"Model type: {info['dtype']}")
print(f"Quantized: {info['quantized']}")

# Context manager for automatic cleanup
with oe.load("runwayml/stable-diffusion-v1-5") as pipe:
    image = pipe.infer("a serene mountain landscape")
    # Model automatically unloaded when exiting context

Audio Models

# Speech-to-text with Whisper
oe.load("openai/whisper-base")
transcription = oe.infer("path/to/audio.wav")
print(transcription)  # "Hello, this is the transcribed audio"

# Text-to-speech (OpenVINO 2025.2+)
oe.load("microsoft/speecht5_tts")
audio = oe.infer("Hello world!")
# Save or play the generated audio

Memory Management

OpenVINO-Easy provides flexible memory management for production applications:

# Method 1: Explicit unload
oe.load("large-model")
result = oe.infer(data)
oe.unload()  # Free memory immediately

# Method 2: Context manager (recommended)
with oe.load("large-model") as pipe:
    result = pipe.infer(data)
    # Model automatically unloaded when exiting

# Method 3: Multiple model switching  
oe.load("text-model")
result1 = oe.infer("Hello world")
oe.unload()

oe.load("image-model")
result2 = oe.infer(image_data)
oe.unload()

# Check if model is still loaded
if oe.is_loaded():
    result = oe.infer(data)
else:
    print("Model has been unloaded")

Model Management & Discovery

OpenVINO-Easy provides comprehensive model management capabilities:

# Search for models on Hugging Face Hub
results = oe.models.search("stable diffusion", limit=5, model_type="image")
for model in results:
    print(f"{model['id']}: {model['downloads']:,} downloads")

# Get detailed model information
info = oe.models.info("microsoft/DialoGPT-medium")
print(f"Local: {info['local']}, Remote: {info['remote']}")
print(f"Requirements: {info['requirements']['min_memory_mb']} MB")

# Install models without loading them
result = oe.models.install("runwayml/stable-diffusion-v1-5", dtype="fp16")
print(f"Installed: {result['size_mb']:.1f} MB")

# Validate model integrity
results = oe.models.validate()
print(f"Validation: {results['passed']}/{results['validated']} models valid")

# Benchmark all installed models
results = oe.models.benchmark_all()
best = results['summary']['fastest_model']
print(f"Fastest model: {best['id']} ({best['fps']:.1f} FPS)")

Model Storage & Cache Management

OpenVINO-Easy uses a clean, Ollama-style directory structure:

# Check where models are stored
print("Models directory:", oe.models.dir())
# Windows: C:\Users\username\AppData\Local\openvino-easy\models\
# Linux/Mac: ~/.openvino-easy/models/

# List all cached models
models_list = oe.models.list()
for model in models_list:
    print(f"{model['name']}: {model['size_mb']:.1f} MB")

# Check cache usage
cache_info = oe.cache.size()
print(f"Total cache size: {cache_info['total_size_mb']:.1f} MB")
print(f"Models: {cache_info['model_count']}")

# Clean up temporary files only (keeps models)
oe.cache.clear()

# Remove a specific model (exact name required for safety)
result = oe.models.remove("microsoft--DialoGPT-medium--fp16--a1b2c3d4")
print(result)  # Shows what was removed

# Clear everything including models (requires confirmation)
result = oe.models.clear()  # Shows safety warning, requires confirm=False
result = oe.models.clear(confirm=False)  # Actually performs deletion

# Clear temp cache only (safe)
oe.cache.clear()

# Clear both temp cache and models (dangerous, requires confirmation)
oe.cache.clear(models=True)  # Shows safety warning
oe.cache.clear(models=True, confirm=False)  # Actually performs deletion

Directory Structure:

~/.openvino-easy/                    # Linux/Mac
C:\Users\user\AppData\Local\openvino-easy\  # Windows
├── models/                          # Downloaded/converted models (permanent)
│   ├── microsoft--DialoGPT-medium--fp16--a1b2c3d4/
│   └── openai--whisper-base--int8--e5f6g7h8/
├── cache/                           # Temporary conversion files
└── config/                          # User settings

Environment Override:

# Custom models directory
export OE_MODELS_DIR="/shared/ai-models"
# or
OE_MODELS_DIR="/shared/ai-models" python app.py

🔧 Command Line Interface

# Text inference
oe run "microsoft/DialoGPT-medium" --prompt "Hello there"

# Audio inference (speech-to-text)
oe run "openai/whisper-base" --input-file "audio.wav"

# Image generation
oe run "runwayml/stable-diffusion-v1-5" --prompt "a beautiful sunset"

# Benchmark with latest NPU precision
oe bench "runwayml/stable-diffusion-v1-5" --dtype fp16-nf4

# System diagnostics
oe doctor

# List available devices
oe devices

# Enhanced NPU diagnostics (Arrow Lake/Lunar Lake detection)
oe npu-doctor

# Cache management
oe cache list              # List cached models
oe cache size              # Show cache usage
oe cache remove <model>    # Remove specific model (with confirmation)
oe cache clear             # Clear temp cache only (safe)
oe cache clear --models    # Clear all models (DANGEROUS - requires confirmation)
oe cache clear --models --force  # Override safety (VERY DANGEROUS)

# Advanced model management
oe models search "stable diffusion" --limit 5  # Search HuggingFace Hub
oe models info microsoft/DialoGPT-medium       # Get model details
oe models install runwayml/stable-diffusion-v1-5 --dtype fp16  # Install model
oe models validate         # Validate all models
oe models benchmark        # Benchmark all installed models

🏗️ Architecture

OpenVINO-Easy wraps OpenVINO's API:

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Your Code     │    │  OpenVINO-Easy   │    │   OpenVINO      │
│                 │    │                  │    │                 │
│ oe.load(...)    │───▶│ • Model Loading  │───▶│ • IR Conversion │
│ oe.infer(...)   │    │ • Device Select  │    │ • Compilation   │
│ oe.benchmark()  │    │ • Preprocessing  │    │ • Inference     │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Key Features

• Device Selection: Chooses NPU → GPU → CPU based on availability
• Model Loading: Supports Hugging Face, ONNX, and OpenVINO IR formats
• Conversion: Converts models to OpenVINO IR format
• INT8 Quantization: Quantization with NNCF for faster inference
• Benchmarking: Performance metrics and timing
• Caching: SHA-256 based model caching for fast re-loading
• Memory Management: Explicit unload() and context manager support
• Hardware Diagnostics: Tools for troubleshooting device issues

🤖 Supported Models

Text Models

• Conversational: DialoGPT, BlenderBot, ChatGLM
• Text Generation: GPT-2, GPT-J, OPT, BLOOM
• Question Answering: BERT, RoBERTa, DeBERTa
• Text Classification: DistilBERT, ALBERT

Vision Models

• Image Generation: Stable Diffusion, DALL-E 2
• Object Detection: YOLO, SSD, RetinaNet
• Image Classification: ResNet, EfficientNet, Vision Transformer
• Segmentation: U-Net, DeepLab, Mask R-CNN

Audio Models

• Speech Recognition: Whisper, Wav2Vec2, WavLM
• Text-to-Speech: SpeechT5, Bark (coming soon)
• Audio Classification: Hubert, Audio Transformers

Multimodal Models

• Vision-Language: CLIP, BLIP, LLaVA
• Image Captioning: BLIP-2, GIT, OFA

🚀 Performance

Performance benchmarks:

Model	Hardware	Throughput	Latency
Stable Diffusion 1.5	Intel Core Ultra 7 Lunar Lake (NPU)	2.3+ img/s	420ms
Stable Diffusion 1.5	Intel Core Ultra 7 Arrow Lake (NPU)	2.2+ img/s	450ms
Stable Diffusion 1.5	Intel Core Ultra 7 (1st gen NPU)	1.8 img/s	556ms
Stable Diffusion 1.5	Intel Arc A770 (GPU)	1.6 img/s	625ms
Stable Diffusion 1.5	Intel Core i7-13700K (CPU)	0.4 img/s	2.5s
DialoGPT-medium	Intel Core Ultra 7 Lunar Lake (NPU)	50+ tok/s	20ms
DialoGPT-medium	Intel Core Ultra 7 Arrow Lake (NPU)	48+ tok/s	21ms
DialoGPT-medium	Intel Core Ultra 7 (1st gen NPU)	40 tok/s	25ms
DialoGPT-medium	Intel Arc A770 (GPU)	38 tok/s	26ms
DialoGPT-medium	Intel Core i7-13700K (CPU)	12 tok/s	83ms

Benchmarks with FP16-NF4 precision on Arrow Lake/Lunar Lake NPUs (OpenVINO 2025.2+)

🔬 Text Processing Details

OpenVINO-Easy handles text preprocessing automatically:

# For text models, tokenization is automatic
pipe = oe.load("microsoft/DialoGPT-medium")

# Multiple input formats supported:
response = pipe.infer("Hello!")                    # String input
response = pipe.infer(["Hello!", "How are you?"])  # Batch input
response = pipe.infer({"text": "Hello!"})          # Dict input

Tokenization Strategy:

1. HuggingFace Models: Uses transformers.AutoTokenizer with model-specific settings
2. ONNX Models: Attempts to infer tokenizer from model metadata
3. OpenVINO IR: Falls back to basic text preprocessing
4. Custom Models: Provides hooks for custom tokenization

🧪 Development & Testing

Modern Python Packaging (Recommended)

# Install in editable mode with development dependencies
pip install -e ".[dev]"

# Or install specific extras for testing
pip install -e ".[full,dev]"  # Full OpenVINO + dev tools

Comprehensive Testing Framework

OpenVINO-Easy includes a robust testing framework with multiple test categories:

# Quick tests (unit tests only, fast)
python test_runner.py --mode fast

# All tests except slow ones
python test_runner.py --mode full

# Integration tests with real OpenVINO models
python test_runner.py --mode integration

# End-to-end tests with real HuggingFace models (requires internet)
python test_runner.py --mode e2e

# Performance regression testing
python test_runner.py --mode performance

# Model compatibility validation
pytest tests/test_model_compatibility.py -v

# Cache management and safety tests
pytest tests/test_model_management.py -v

# CLI functionality tests
pytest tests/test_cli_models.py -v

# Run with coverage
python test_runner.py --mode coverage

Quality Assurance Features

Performance Regression Testing

# Automated performance baselines
from tests.test_performance_regression_enhanced import PerformanceRegression

tester = PerformanceRegression()
test = PerformanceTest(
    model_id="microsoft/DialoGPT-medium",
    tolerance_percent=15.0  # Allow 15% regression
)

results = tester.run_performance_test(test)
if results['regressions']:
    print("Performance regressions detected!")

Model Compatibility Validation

# Automated compatibility testing across devices/precisions
from tests.test_model_compatibility import ModelCompatibilityValidator

validator = ModelCompatibilityValidator()
result = validator.validate_model_compatibility("runwayml/stable-diffusion-v1-5")

if not result['overall_compatible']:
    print(f"Compatibility issues: {result['issues']}")

Enhanced Error Recovery

# Automatic device fallback and retry logic
oe.load(
    "microsoft/DialoGPT-medium",
    device_preference=["NPU", "GPU", "CPU"],
    retry_on_failure=True,
    fallback_device="CPU"
)
# Automatically tries NPU -> GPU -> CPU -> CPU with default config

Test Categories

Test Type	Command	Purpose
Unit Tests	pytest tests/ -m "not slow and not integration"	Core functionality
Integration Tests	pytest tests/ -m "integration"	Real model loading
Performance Tests	pytest tests/ -m "performance"	Regression detection
Compatibility Tests	pytest tests/ -m "compatibility"	Device/model validation
End-to-End Tests	pytest tests/test_e2e_real_models.py	Full workflows
CLI Tests	pytest tests/test_cli*.py	Command-line interface
Safety Tests	pytest tests/test_model_management.py	Security validation

Development Workflow

# Format code
black oe/ tests/
isort oe/ tests/

# Type checking
mypy oe/

# Run all quality checks
python test_runner.py --mode full
pytest tests/test_model_compatibility.py -x
pytest tests/test_performance_regression_enhanced.py -x

📚 Examples

Check out the examples/ directory:

• Stable Diffusion Notebook: Image generation with automatic optimization
• Text Generation: Conversational AI with DialoGPT
• ONNX Models: Loading and running ONNX models
• Custom Models: Integrating your own models

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

📄 License

Apache License 2.0 - see LICENSE for details.

🙏 Acknowledgments

• Intel OpenVINO Team for the inference engine
• Hugging Face for the transformers ecosystem
• ONNX Community for the model format standards