video-comprehension-score 1.0.0

A Comprehensive Python Library for Narrative Similarity Evaluation between two very long descriptions

📄 Research Paper · 📓 Interactive Notebook

🤔 What is VCS?

Recent advances in Large Video Language Models (LVLMs) have significantly enhanced automated video understanding, enabling detailed, long-form narratives of complex video content. However, accurately evaluating whether these models genuinely comprehend the video's narrative—its events, entities, interactions, and chronological coherence—remains challenging.

Why Existing Metrics Fall Short:

• N-gram Metrics (e.g., BLEU, ROUGE, CIDEr): Primarily measure lexical overlap, penalizing valid linguistic variations and inadequately evaluating narrative chronology.

• Embedding-based Metrics (e.g., BERTScore, SBERT): Improve semantic sensitivity but struggle with extended context, detailed content alignment, and narrative sequencing.

• LLM-based Evaluations: Often inconsistent, lacking clear criteria for narrative structure and chronology assessments.

Moreover, traditional benchmarks largely rely on question-answering tasks, which only test isolated events or entities rather than holistic video comprehension. A model answering specific questions correctly does not necessarily demonstrate understanding of the overall narrative or the intricate interplay of events.

Introducing VCS (Video Comprehension Score):

VCS is a Python library specifically designed to overcome these challenges by evaluating narrative comprehension through direct comparison of extensive, detailed video descriptions generated by LVLMs against human-written references. Unlike traditional metrics, VCS assesses whether models capture the overall narrative structure, event sequencing, and thematic coherence, not just lexical or isolated semantic matches.

Core Components of VCS:

• 🌍 Global Alignment Score (GAS): Captures overall thematic alignment, tolerating stylistic variations without penalizing valid linguistic differences.

• 🎯 Local Alignment Score (LAS): Checks detailed semantic correspondence at a chunk-level, allowing minor descriptive variations while penalizing significant inaccuracies or omissions.

• 📖 Narrative Alignment Score (NAS): Evaluates chronological consistency, balancing the need for both strict event sequencing and permissible narrative flexibility.

Initially developed for evaluating video comprehension by comparing generated and human-written video narratives, VCS is versatile enough for broader applications, including document-level narrative comparisons, analysis of extensive narrative content, and various other narrative similarity tasks.

🚀 Key Applications

🌟 Key Features

Explore the comprehensive capabilities that make VCS a powerful narrative evaluation toolkit. To understand these features in detail, read our research paper, then visit our interactive playground to see them in action.

⚡ Getting Started

Welcome to VCS Metrics! This guide will walk you through everything you need to start analyzing narrative similarity between texts. We'll cover installation, setup, and your first VCS analysis step by step.

📦 Step 1: Installation

Choose the installation method that fits your needs:

pip install video-comprehension-score

!pip install video-comprehension-score

git clone https://github.com/hdubey-debug/vcs.git
cd vcs
pip install -e ".[dev]"
pre-commit install

!git clone https://github.com/hdubey-debug/vcs.git
%cd vcs
!pip install -e ".[dev]"
!pre-commit install

🛠️ System Requirements

Before installing VCS, make sure your system meets these requirements:

🐍 Python Required: Python 3.10 or higher VCS Metrics uses modern Python features and requires Python 3.10+. We recommend Python 3.11+ for optimal performance.

🔥 PyTorch Required: PyTorch 1.9.0+ VCS needs PyTorch but doesn't install it automatically to avoid conflicts. Get it from the official PyTorch website. 💡 Pro Tip: In Google Colab, PyTorch is pre-installed!

🔧 Step 2: Prepare Your Functions

Now that VCS is installed, you need to define two functions before you can use the VCS API. Here's how VCS works:

📋 VCS API Overview

from vcs import compute_vcs_score

result = compute_vcs_score(
    reference_text="Your reference text here",
    generated_text="Your generated text here", 
    segmenter_fn=your_segmenter_function,        # ← You provide this
    embedding_fn_las=your_embedding_function,    # ← You provide this  
    embedding_fn_gas=your_embedding_function,    # ← You provide this
    return_all_metrics=True
)

print(f"VCS Score: {result['VCS']:.4f}")

As you can see, VCS requires two custom functions from you. Let's understand what each should do:

def your_segmenter(text: str) -> List[str]:
    # Your implementation here
    return list_of_text_segments

def your_embedder(texts: List[str]) -> torch.Tensor:
    # Your implementation here  
    return tensor_of_embeddings

🌟 Author Recommendations (2025)

For best results, we recommend these state-of-the-art models:

🔪 Segmentation Champion 🏆 Recommended: Segment Any Text (SAT) ✨ Why we recommend SAT: 🎯 State-of-the-art segmentation accuracy ⚡ Intelligent boundary detection 🧠 Context-aware text splitting 🔬 Research-grade performance 🔗 Repository: github.com/segment-any-text/wtpsplit

🧠 Embedding Powerhouse 🥇 Recommended: nv-embed-v2 🌟 Why we recommend nv-embed-v2: 📊 Top performer on MTEB Leaderboard 🚀 Superior semantic understanding 💪 Robust multilingual support ⚡ Excellent for VCS analysis 🔗 Model: nvidia/NV-Embed-v2

💻 Step 3: Run Your First VCS Analysis

Now let's see VCS in action with a complete working example:

# Fix import path issue if running from vcs/ root directory
import sys
import os
if os.path.basename(os.getcwd()) == 'vcs' and os.path.exists('src/vcs'):
    sys.path.insert(0, 'src')
    print("🔧 Fixed import path for development directory")

# Test the installation
try:
    import vcs
    print("✅ VCS package imported successfully!")
    
    # Test main function availability
    if hasattr(vcs, 'compute_vcs_score'):
        print("✅ Main function 'compute_vcs_score' is available!")
    else:
        print("⚠️ Main function not found - there might be an installation issue")
        
    # Try to get version
    try:
        print(f"📦 Version: {vcs.__version__}")
    except AttributeError:
        print("📦 Version: Unable to determine (this is normal for development installs)")
        
except ImportError as e:
    print(f"❌ Import failed: {e}")
    print("💡 Make sure you:")
    print("   1. Installed VCS correctly: pip install -e .[dev]")
    print("   2. Restarted your notebook kernel") 
    print("   3. You're NOT in the root vcs/ directory (this causes import conflicts)")

# Import required libraries
import torch
from typing import List

# Define lightweight segmenter function
def simple_segmenter(text: str) -> List[str]:
    """
    Simple sentence segmenter using period splitting.
    
    Args:
        text: Input text to segment
        
    Returns:
        List of text segments
    """
    # Split by periods and clean up
    segments = [s.strip() for s in text.split('.') if s.strip()]
    return segments

# Define lightweight embedding function using sentence-transformers
def lightweight_embedding_function(texts: List[str]) -> torch.Tensor:
    """
    Lightweight embedding function using sentence-transformers.
    
    Args:
        texts: List of text segments to embed
        
    Returns:
        PyTorch tensor of shape (len(texts), embedding_dim)
    """
    try:
        from sentence_transformers import SentenceTransformer
        
        # Use a lightweight model (only downloads ~80MB)
        model = SentenceTransformer('all-MiniLM-L6-v2')
        
        # Generate embeddings
        embeddings = model.encode(texts)
        return torch.tensor(embeddings, dtype=torch.float32)
        
    except ImportError:
        print("⚠️ sentence-transformers not found. Installing...")
        import subprocess
        import sys
        subprocess.check_call([sys.executable, "-m", "pip", "install", "sentence-transformers"])
        
        # Try again after installation
        from sentence_transformers import SentenceTransformer
        model = SentenceTransformer('all-MiniLM-L6-v2')
        embeddings = model.encode(texts)
        return torch.tensor(embeddings, dtype=torch.float32)

# Example texts
reference_text = """
The quick brown fox jumps over the lazy dog.
It was a beautiful sunny day in the forest.
The fox was looking for food for its family.
"""

generated_text = """
A brown fox jumped over a sleeping dog.
The weather was nice and sunny in the woods.
The fox needed to find food for its cubs.
"""

# Compute VCS score
print("🧠 Computing VCS score...")
try:
    result = vcs.compute_vcs_score(
        reference_text=reference_text,
        generated_text=generated_text,
        segmenter_fn=simple_segmenter,
        embedding_fn_las=lightweight_embedding_function,
        embedding_fn_gas=lightweight_embedding_function,
        return_all_metrics=True,
        return_internals=True
    )
    
    print("🎯 VCS Results:")
    print(f"VCS Score: {result['VCS']:.4f}")
    print(f"GAS Score: {result['GAS']:.4f}")
    print(f"LAS Score: {result['LAS']:.4f}")
    print(f"NAS Score: {result['NAS']:.4f}")
    print("✅ VCS is working correctly!")
    
    # Generate visualization (optional)
    if 'internals' in result:
        try:
            fig = vcs.visualize_metrics_summary(result['internals'])
            print("📊 Visualization generated successfully!")
            # fig.show()  # Uncomment to display
        except Exception as viz_error:
            print(f"⚠️ Visualization failed (this is normal in some environments): {viz_error}")
    
except Exception as e:
    print(f"❌ Error running VCS: {e}")
    print("💡 Make sure PyTorch is installed and try restarting your kernel")

⚙️ Advanced Configuration

Once you're comfortable with the basics, you can fine-tune VCS behavior for your specific use case:

# 🎯 Comprehensive configuration with all features enabled
result = compute_vcs_score(
    reference_text=ref_text,
    generated_text=gen_text,
    segmenter_fn=segmenter,
    embedding_fn_las=embedder,
    embedding_fn_gas=embedder,
    chunk_size=2,                  # Group segments
    context_cutoff_value=0.7,      # Higher threshold
    context_window_control=3.0,    # Tighter windows
    lct=1,                         # Some reordering OK
    return_all_metrics=True,       # Get detailed breakdown
    return_internals=True          # Get visualization data
)

❓ Frequently Asked Questions

🏗️ Project Structure

vcs/
├── 📁 src/vcs/                  # Main package source code
│   ├── 📄 __init__.py           # Package initialization
│   ├── 📄 scorer.py             # Main VCS API entry point
│   ├── 📄 _config.py            # Configuration settings
│   ├── 📁 _metrics/             # Core VCS metrics implementations
│   │   ├── 📁 _gas/             # Global Alignment Score
│   │   ├── 📁 _las/             # Local Alignment Score  
│   │   ├── 📁 _nas/             # Narrative Alignment Score with components
│   │   │   └── 📁 _nas_components/  # Distance NAS, Line NAS, Regularize NAS
│   │   └── 📁 _vcs/             # Combined VCS computation
│   ├── 📁 _visualize_vcs/       # Comprehensive visualization suite
│   │   ├── 📁 _similarity_matrix/  # Similarity matrix visualizations
│   │   ├── 📁 _best_match/      # Best match analysis plots
│   │   ├── 📁 _distance_nas/    # Distance-based NAS visualizations
│   │   ├── 📁 _line_nas/        # Line-based NAS visualizations
│   │   ├── 📁 _mapping_windows/ # Context window visualizations
│   │   ├── 📁 _metrics_summary/ # Overall metrics summary plots
│   │   ├── 📁 _pdf_report/      # PDF report generation
│   │   ├── 📁 _text_chunks/     # Text chunk visualizations
│   │   ├── 📁 _window_regularizer/ # Window regularizer plots
│   │   ├── 📁 _las/             # LAS-specific visualizations
│   │   └── 📁 _config/          # Visualization configuration
│   ├── 📁 _segmenting/          # Text segmentation utilities
│   ├── 📁 _matching/            # Optimal text matching algorithms
│   ├── 📁 _mapping_windows/     # Context window management
│   └── 📁 _utils/               # Helper utilities
├── 📁 docs/                     # Documentation and interactive demos
│   ├── 📄 index.html            # Main documentation website
│   ├── 📁 pages/                # Documentation pages
│   │   ├── 📄 api.html          # API reference
│   │   ├── 📄 playground.html   # Interactive playground
│   │   └── 📄 example.html      # Usage examples
│   ├── 📁 widgets/              # Interactive visualization widgets
│   ├── 📁 sphinx/               # Sphinx documentation source
│   └── 📁 assets/               # Documentation assets (CSS, JS, videos)
├── 📁 .github/                  # GitHub configuration
│   ├── 📁 assets/               # README assets (images, gifs)
│   ├── 📁 scripts/              # GitHub automation scripts
│   └── 📁 workflows/            # CI/CD automation pipelines
│       ├── 📄 test.yml          # Continuous testing
│       ├── 📄 publish.yml       # Package publishing
│       └── 📄 deploy-docs.yml   # Documentation deployment
├── 📄 pyproject.toml           # Package configuration & dependencies
├── 📄 CONTRIBUTING.md          # Development contribution guide
├── 📄 DEPLOYMENT.md            # Release and deployment guide
├── 📄 CHANGELOG.md             # Version history and changes
├── 📄 MANIFEST.in              # Package manifest
├── 📄 tag_version.py           # Version tagging script
├── 📄 LICENSE                  # MIT license
└── 📄 README.md                # This documentation

🚀 Development & Contributing

We welcome contributions to VCS Metrics! Whether you're fixing bugs, adding features, or improving documentation, here's how to get started.

🛠️ Quick Development Setup

# 1. Clone and setup
git clone https://github.com/hdubey-debug/vcs.git
cd vcs
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# 2. Install development dependencies
pip install -e .[dev]

# 3. Create your feature branch
git checkout -b feature/your-feature-name

# 4. Make your changes
# Edit files in src/vcs/
# Add tests if needed
# Update docs if necessary

# 5. Run quality checks
black src/ && isort src/ && flake8 src/ && mypy src/

# 6. Commit with semantic format
git commit -m "minor: add new awesome feature"

# 7. Push and create PR
git push origin feature/your-feature-name

📋 Contribution Workflow

💡 Semantic Commit Format

We use semantic commits for automatic version bumping:

🔧 Automated Testing & CI/CD

Our comprehensive CI/CD pipeline ensures code quality and reliability on every commit:

📖 Detailed Guides

For comprehensive information about contributing and development:

🤝 Getting Help

📚 Citation

If you use VCS Metrics in your research, please cite:

@software{vcs_metrics_2024,
  title = {VCS Metrics: Video Comprehension Score for Text Similarity Evaluation},
  author = {Dubey, Harsh and Ali, Mukhtiar and Mishra, Sugam and Pack, Chulwoo},
  year = {2024},
  institution = {South Dakota State University},
  url = {https://github.com/hdubey-debug/vcs},
  note = {Python package for narrative similarity evaluation}
}

🤖 CLIP-CC Ecosystem Integration

VCS is designed to work seamlessly with CLIP-CC Dataset for comprehensive video understanding evaluation.

🏆 Meet Our Contributors

Harsh Dubey Lead Developer & Research Scientist South Dakota State University Commits Lines Files 2 49K 171 📋 Key Work: • VCS Algorithm Architecture • Visualization Engine • LAS, GAS, and NAS Metrics

🤖 Automated Contributors

Contributor	Role	Contributions	Badge
🤖 GitHub Actions	CI/CD Automation	Clean history setup

📊 Contribution Analytics

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🌟 Made with ❤️ by the VCS Team

Authors: Harsh Dubey, Mukhtiar Ali, Sugam Mishra, and Chulwoo Pack
Institution: South Dakota State University
Year: 2024

⭐ Star this repo • 🐛 Report Bug • 💡 Request Feature • 💬 Community Q&A