VideoChain

Edge-optimized multimodal RAG framework for video understanding — transforms raw footage into a structured, queryable knowledge base.

Overview

VideoChain is a lightweight, modular framework that combines computer vision, speech recognition, and LLM reasoning into a unified late-fusion pipeline. Designed to run on consumer-grade GPUs (tested on NVIDIA RTX 3050), it carefully schedules VRAM across concurrent vision and language inference — making on-device video intelligence practical without cloud dependency.

Core Pipeline

Video Input → Frame Extraction → Vision Inference → Audio Transcription → Fusion Engine → Knowledge Base → LLM Query

Key Capabilities

Adaptive Keyframe Extraction

Gaussian-blurred frame differencing filters transient noise and isolates semantically significant motion events, reducing redundant frame processing by discarding visually similar consecutive frames.

Multimodal Data Alignment

Visual labels from MobileNetV3 and Whisper-generated transcripts are synchronized via timestamp, producing a unified timeline for downstream retrieval and reasoning.

Domain-Agnostic Design

Modular loader and processor interfaces allow straightforward adaptation to security, retail analytics, education, and personal content search — without restructuring the core pipeline.

Edge-First Optimization

Concurrent vision and LLM inference with careful VRAM scheduling. Validated on RTX 3050 (4 GB). No cloud inference dependency for core pipeline execution.

Installation

# Clone the repository
git clone https://github.com/rahulsiiitm/videochain
cd videochain

# Install in editable mode (recommended for development)
pip install -e .

⚠️ Requirement: NVIDIA drivers and CUDA 12.1 are required for GPU-accelerated inference. CPU-only execution is supported but significantly slower for vision workloads.

Quick Start

1 — Build a knowledge base

Analyze a video file and generate a structured JSON knowledge base:

videochain-analyze --input sample.mp4

Output: knowledge_base.json

2 — Train a custom vision model

Fine-tune the vision classifier on a domain-specific dataset:

videochain-train --epochs 15 --batch-size 16

Place labeled training images under data/train/ before running. Class subdirectory names become label strings in the knowledge base.

3 — Query the knowledge base

Use Ollama (local) or Gemini API (cloud) to issue natural language queries over the generated knowledge base.

System Architecture

VideoChain follows a late-fusion architecture — each modality is processed independently before being merged at the knowledge-base level. This decouples model upgrade paths and allows per-modality optimization.

Knowledge Base Schema

Each event entry in knowledge_base.json follows this structure:

{
  "timestamp": "00:01:23",
  "visual": ["person", "running"],
  "audio": "Someone is running across the hallway",
  "confidence": 0.91,
  "frame_index": 2490
}

Project Structure

videochain/
├── core/            # Fusion engine, LLM query interface, KB I/O
├── loaders/         # Video frame extraction, audio separation
├── processors/      # MobileNetV3 vision, Whisper audio inference
├── scripts/         # Training utilities, dataset prep helpers
├── data/
│   └── train/       # Class-labeled training images (one dir per class)
└── pyproject.toml   # Dependencies, CLI entry points, metadata

Tech Stack

Use Cases

Roadmap

• Real-time streaming pipeline — live ingestion and indexing with low-latency event detection
• Vector database integration — FAISS or Chroma backends for semantic similarity search
• Advanced temporal reasoning — event co-occurrence detection, causal chain inference, multi-clip reasoning
• Query dashboard — browser-based UI for video playback, timeline visualization, and KB exploration

Contributing

Contributions, issues, and feature requests are welcome. Open a GitHub issue or submit a pull request.

Author

Rahul Sharma — B.Tech CSE, IIIT Manipur

License

Distributed under the MIT License.