sharingan-core 3.0.0

OpenCV for Semantic Video Understanding with Temporal Reasoning

👁️ Sharingan - Semantic Video Understanding

Sharingan is a lightweight Python library for semantic video understanding with temporal reasoning. It combines vision-language models (CLIP, SmolVLM) with temporal analysis to understand video content at a deep semantic level.

---

✨ Features

• 🎬 Semantic Video Processing – Understand video content beyond pixels
• 🔍 Natural Language Queries – Search videos using text descriptions
• 🤖 AI Chat – Conversational interface with Qwen2.5-0.5B
• ⚡ Temporal Reasoning – Cross-frame attention and memory tokens
• 🎯 Event Detection – Automatically identify key moments
• 💾 Efficient Storage – 130x compression with Int8 quantization
• 🌐 Web UI – Beautiful Flask-based interface
• 🚀 Fast Processing – Batch processing and GPU acceleration

---

You can read the Author Note, check out the Architecture, and see the Contributing Guidelines on GitHub.

---

🚀 Quick Start

Installation

pip install sharingan-core

# Optional: GPU acceleration
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118

# Optional: AI chat
pip install transformers bitsandbytes accelerate

Basic Usage

from sharingan import VideoProcessor

processor = VideoProcessor(
    vlm_model='clip',  # or 'smolvlm'
    device='auto'
)

results = processor.process('video.mp4')

matches = processor.query('person speaking')
for match in matches:
    print(f"Found at {match.timestamp}s - {match.confidence:.2%}")

response = processor.chat('What happens in this video?')
print(response)

Web UI

python -m sharingan.cli ui

Or programmatically:

from sharingan.ui import run_ui
run_ui(port=5000, open_browser=True)

---

📖 Documentation

Vision Models

• CLIP – Fast semantic embeddings; memory ~400MB
• SmolVLM-500M – Detailed frame descriptions; memory ~538MB (8-bit quantized)

Processing Options

processor = VideoProcessor(
    vlm_model='clip',
    device='auto',
    target_fps=5.0,
    enable_temporal=True,
    enable_tracking=False
)

Query Options

results = processor.query('person speaking', top_k=5)
response = processor.chat('Describe main events', use_llm=True)

---

🎯 Use Cases

• Video Search – Find moments using natural language
• Content Moderation – Detect inappropriate content
• Video Summarization – Auto summaries
• Accessibility – Descriptions for visually impaired
• Research – Analyze video datasets at scale

---

🔧 Advanced Features

Temporal Reasoning

• Cross-Frame Gating – Learns important frames
• Memory Tokens – Maintains context across video
• Temporal Attention – Understand relationships between frames

Efficient Storage

• 5-min video: ~2.3MB (vs 300MB raw)
• Fast cache loading
• Minimal quality loss for search

Event Detection

• Scene changes
• Motion patterns
• Content transitions

---

📊 Performance

Model	Speed	Memory	Quality
CLIP	⚡⚡⚡	400MB	Good
SmolVLM	⚡⚡	538MB	Excellent

Tested on NVIDIA RTX 3050 (4GB VRAM)

---

🤝 Contributing

Contributions welcome! Please submit a PR.

📄 License

MIT License – see LICENSE file.

🙏 Acknowledgments

• OpenAI CLIP
• SmolVLM
• Qwen2.5

📧 Contact

Open an issue on GitHub for support.

---

Made with ☕ & ❤️