video-comprehension-score 1.0.0

Video Comprehension Score (VCS) - A comprehensive metric for evaluating narrative similarity between reference and generated text

A Comprehensive Python Library for Narrative Similarity Evaluation between two very long descriptions

📄 Research Paper · 📓 Interactive Notebook

---

🤔 What is VCS?

Recent advances in Large Video Language Models (LVLMs) have significantly enhanced automated video understanding, enabling detailed, long-form narratives of complex video content. However, accurately evaluating whether these models genuinely comprehend the video's narrative—its events, entities, interactions, and chronological coherence—remains challenging.

Why Existing Metrics Fall Short:

• N-gram Metrics (e.g., BLEU, ROUGE, CIDEr): Primarily measure lexical overlap, penalizing valid linguistic variations and inadequately evaluating narrative chronology.

• Embedding-based Metrics (e.g., BERTScore, SBERT): Improve semantic sensitivity but struggle with extended context, detailed content alignment, and narrative sequencing.

• LLM-based Evaluations: Often inconsistent, lacking clear criteria for narrative structure and chronology assessments.

Moreover, traditional benchmarks largely rely on question-answering tasks, which only test isolated events or entities rather than holistic video comprehension. A model answering specific questions correctly does not necessarily demonstrate understanding of the overall narrative or the intricate interplay of events.

Introducing VCS (Video Comprehension Score):

VCS is a Python library specifically designed to overcome these challenges by evaluating narrative comprehension through direct comparison of extensive, detailed video descriptions generated by LVLMs against human-written references. Unlike traditional metrics, VCS assesses whether models capture the overall narrative structure, event sequencing, and thematic coherence, not just lexical or isolated semantic matches.

Core Components of VCS:

• 🌍 Global Alignment Score (GAS): Captures overall thematic alignment, tolerating stylistic variations without penalizing valid linguistic differences.

• 🎯 Local Alignment Score (LAS): Checks detailed semantic correspondence at a chunk-level, allowing minor descriptive variations while penalizing significant inaccuracies or omissions.

• 📖 Narrative Alignment Score (NAS): Evaluates chronological consistency, balancing the need for both strict event sequencing and permissible narrative flexibility.

Initially developed for evaluating video comprehension by comparing generated and human-written video narratives, VCS is versatile enough for broader applications, including document-level narrative comparisons, analysis of extensive narrative content, and various other narrative similarity tasks.

🚀 Key Applications

🎯 Transform Your Narrative Analysis Across Every Domain

🎬 Video Intelligence Evaluate video language models' narrative comprehension	📄 Document Similarity Compare semantic alignment between long-form documents	📖 Story Analysis Measure narrative similarity between different stories
🎓 Academic Research Detect conceptual plagiarism and idea overlap	📝 Paragraph Analysis Evaluate narrative coherence within text sections	🎯 Short Caption Evaluation Evaluate short captions and brief descriptions

---

🌟 Key Features

Explore the comprehensive capabilities that make VCS a powerful narrative evaluation toolkit. To understand these features in detail, read our research paper, then visit our interactive playground to see them in action.

🧮 Comprehensive Metric Suite Computes VCS along with detailed breakdowns: GAS (global thematic similarity), LAS with precision/recall components, and NAS with distance-based and line-based sub-metrics. Access all internal calculations including penalty systems, mapping windows, and alignment paths.	📊 Advanced Visualization Engine 11 specialized visualization functions including similarity heatmaps, alignment analysis, best-match visualizations, narrative flow diagrams, and precision/recall breakdowns. Each metric component can be visualized with publication-quality plots.
📋 Professional PDF Reports Generate comprehensive multi-page PDF reports with all metrics, visualizations, and analysis details. Supports both complete reports and customizable selective reports. Professional formatting suitable for research publications.	⚙️ Flexible Configuration System Fine-tune evaluation with configurable parameters: chunk sizes, similarity thresholds, context windows, and Local Chronology Tolerance (LCT). Supports custom segmentation and embedding functions for domain-specific applications.

---

⚡ Getting Started

Welcome to VCS Metrics! This guide will walk you through everything you need to start analyzing narrative similarity between texts. We'll cover installation, setup, and your first VCS analysis step by step.

---

📦 Step 1: Installation

Choose the installation method that fits your needs:

🎯 For Most Users Recommended if you just want to use VCS 🖱️ Click to expand installation steps Terminal Installation: pip install video-comprehension-score Jupyter/Colab Installation: !pip install video-comprehension-score ✅ Ready in 30 seconds 🔥 Zero configuration needed ⚡ Instant access to all features

🛠️ For Developers If you want to contribute or modify VCS 🖱️ Click to expand development setup Terminal Installation: git clone https://github.com/hdubey-debug/vcs.git cd vcs pip install -e ".[dev]" pre-commit install Jupyter/Colab Installation: !git clone https://github.com/hdubey-debug/vcs.git %cd vcs !pip install -e ".[dev]" !pre-commit install 🔧 Latest features first 🧪 Testing capabilities 🤝 Contribution ready

---

🛠️ System Requirements

Before installing VCS, make sure your system meets these requirements:

🐍 Python Required: Python 3.10 or higher VCS Metrics uses modern Python features and requires Python 3.10+. We recommend Python 3.11+ for optimal performance.

🔥 PyTorch Required: PyTorch 1.9.0+ VCS needs PyTorch but doesn't install it automatically to avoid conflicts. Get it from the official PyTorch website. 💡 Pro Tip: In Google Colab, PyTorch is pre-installed!

💡 Note: VCS automatically installs dependencies: numpy≥1.20.0, matplotlib≥3.5.0, seaborn≥0.11.0

---

🔧 Step 2: Prepare Your Functions

Now that VCS is installed, you need to define two functions before you can use the VCS API. Here's how VCS works:

📋 VCS API Overview

from vcs import compute_vcs_score

result = compute_vcs_score(
    reference_text="Your reference text here",
    generated_text="Your generated text here", 
    segmenter_fn=your_segmenter_function,        # ← You provide this
    embedding_fn_las=your_embedding_function,    # ← You provide this  
    embedding_fn_gas=your_embedding_function,    # ← You provide this
    return_all_metrics=True
)

print(f"VCS Score: {result['VCS']:.4f}")

As you can see, VCS requires two custom functions from you. Let's understand what each should do:

🔪 Segmenter Function What it does: Splits text into meaningful segments (sentences, paragraphs, etc.) Required signature: def your_segmenter(text: str) -> List[str]: # Your implementation here return list_of_text_segments Arguments: text (str) - Input text to be segmented Returns: List[str] - List of text segments You can use: Any library or model (NLTK, spaCy, custom logic, etc.)

🧠 Embedding Function What it does: Converts text segments into numerical vectors (embeddings) Required signature: def your_embedder(texts: List[str]) -> torch.Tensor: # Your implementation here return tensor_of_embeddings Arguments: texts (List[str]) - List of text segments to embed Returns: torch.Tensor - Tensor of shape (len(texts), embedding_dim) You can use: Any embedding model (sentence-transformers, OpenAI, etc.)

🌟 Author Recommendations (2025)

For best results, we recommend these state-of-the-art models:

⚠️ Note: These recommendations are current as of 2025. Always research the latest SOTA options.

🔪 Segmentation Champion 🏆 Recommended: Segment Any Text (SAT) ✨ Why we recommend SAT: 🎯 State-of-the-art segmentation accuracy ⚡ Intelligent boundary detection 🧠 Context-aware text splitting 🔬 Research-grade performance 🔗 Repository: github.com/segment-any-text/wtpsplit

🧠 Embedding Powerhouse 🥇 Recommended: nv-embed-v2 🌟 Why we recommend nv-embed-v2: 📊 Top performer on MTEB Leaderboard 🚀 Superior semantic understanding 💪 Robust multilingual support ⚡ Excellent for VCS analysis 🔗 Model: nvidia/NV-Embed-v2

💡 Alternative Options: NLTK, spaCy, sentence-transformers, or build your own custom functions!

---

💻 Step 3: Run Your First VCS Analysis

Now let's see VCS in action with a complete working example:

⚡ Performance Notes SOTA models require GPU. For CPU testing, this example uses lightweight alternatives.

🚀 Quick Example - Click to expand complete tutorial

🎯 Complete Working Example

Copy, paste, and run this code to see VCS in action

# Fix import path issue if running from vcs/ root directory
import sys
import os
if os.path.basename(os.getcwd()) == 'vcs' and os.path.exists('src/vcs'):
    sys.path.insert(0, 'src')
    print("🔧 Fixed import path for development directory")

# Test the installation
try:
    import vcs
    print("✅ VCS package imported successfully!")
    
    # Test main function availability
    if hasattr(vcs, 'compute_vcs_score'):
        print("✅ Main function 'compute_vcs_score' is available!")
    else:
        print("⚠️ Main function not found - there might be an installation issue")
        
    # Try to get version
    try:
        print(f"📦 Version: {vcs.__version__}")
    except AttributeError:
        print("📦 Version: Unable to determine (this is normal for development installs)")
        
except ImportError as e:
    print(f"❌ Import failed: {e}")
    print("💡 Make sure you:")
    print("   1. Installed VCS correctly: pip install -e .[dev]")
    print("   2. Restarted your notebook kernel") 
    print("   3. You're NOT in the root vcs/ directory (this causes import conflicts)")

# Import required libraries
import torch
from typing import List

# Define lightweight segmenter function
def simple_segmenter(text: str) -> List[str]:
    """
    Simple sentence segmenter using period splitting.
    
    Args:
        text: Input text to segment
        
    Returns:
        List of text segments
    """
    # Split by periods and clean up
    segments = [s.strip() for s in text.split('.') if s.strip()]
    return segments

# Define lightweight embedding function using sentence-transformers
def lightweight_embedding_function(texts: List[str]) -> torch.Tensor:
    """
    Lightweight embedding function using sentence-transformers.
    
    Args:
        texts: List of text segments to embed
        
    Returns:
        PyTorch tensor of shape (len(texts), embedding_dim)
    """
    try:
        from sentence_transformers import SentenceTransformer
        
        # Use a lightweight model (only downloads ~80MB)
        model = SentenceTransformer('all-MiniLM-L6-v2')
        
        # Generate embeddings
        embeddings = model.encode(texts)
        return torch.tensor(embeddings, dtype=torch.float32)
        
    except ImportError:
        print("⚠️ sentence-transformers not found. Installing...")
        import subprocess
        import sys
        subprocess.check_call([sys.executable, "-m", "pip", "install", "sentence-transformers"])
        
        # Try again after installation
        from sentence_transformers import SentenceTransformer
        model = SentenceTransformer('all-MiniLM-L6-v2')
        embeddings = model.encode(texts)
        return torch.tensor(embeddings, dtype=torch.float32)

# Example texts
reference_text = """
The quick brown fox jumps over the lazy dog.
It was a beautiful sunny day in the forest.
The fox was looking for food for its family.
"""

generated_text = """
A brown fox jumped over a sleeping dog.
The weather was nice and sunny in the woods.
The fox needed to find food for its cubs.
"""

# Compute VCS score
print("🧠 Computing VCS score...")
try:
    result = vcs.compute_vcs_score(
        reference_text=reference_text,
        generated_text=generated_text,
        segmenter_fn=simple_segmenter,
        embedding_fn_las=lightweight_embedding_function,
        embedding_fn_gas=lightweight_embedding_function,
        return_all_metrics=True,
        return_internals=True
    )
    
    print("🎯 VCS Results:")
    print(f"VCS Score: {result['VCS']:.4f}")
    print(f"GAS Score: {result['GAS']:.4f}")
    print(f"LAS Score: {result['LAS']:.4f}")
    print(f"NAS Score: {result['NAS']:.4f}")
    print("✅ VCS is working correctly!")
    
    # Generate visualization (optional)
    if 'internals' in result:
        try:
            fig = vcs.visualize_metrics_summary(result['internals'])
            print("📊 Visualization generated successfully!")
            # fig.show()  # Uncomment to display
        except Exception as viz_error:
            print(f"⚠️ Visualization failed (this is normal in some environments): {viz_error}")
    
except Exception as e:
    print(f"❌ Error running VCS: {e}")
    print("💡 Make sure PyTorch is installed and try restarting your kernel")

📝 Scale Note: This example uses small text for illustration - VCS excels with long-form content! ⚠️ Import Tip: Running from vcs/ root? The example includes an automatic path fix.

---

⚙️ Advanced Configuration

Once you're comfortable with the basics, you can fine-tune VCS behavior for your specific use case:

🎯 Core Parameters 🎛️ Essential Controls: Parameter Default Purpose chunk_size 1 Segment grouping context_cutoff_value 0.6 Similarity threshold context_window_control 4.0 Context window size lct 0 Narrative reordering tolerance

📊 Return All Metrics 🎛️ Control Parameter: Parameter Default Purpose return_all_metrics False Return detailed metric breakdown When set to True, you get: Individual GAS, LAS, NAS scores LAS precision and recall components Distance-based and line-based NAS sub-metrics Complete metric breakdown for analysis

🔍 Return Internals 🎛️ Control Parameter: Parameter Default Purpose return_internals False Return internal computation data When set to True, you get: Similarity matrices and alignment paths Mapping windows and penalty calculations Text chunks and segmentation details All data needed for visualization

🚀 **Example Configuration** # 🎯 Comprehensive configuration with all features enabled result = compute_vcs_score( reference_text=ref_text, generated_text=gen_text, segmenter_fn=segmenter, embedding_fn_las=embedder, embedding_fn_gas=embedder, chunk_size=2, # Group segments context_cutoff_value=0.7, # Higher threshold context_window_control=3.0, # Tighter windows lct=1, # Some reordering OK return_all_metrics=True, # Get detailed breakdown return_internals=True # Get visualization data )

📚 For complete API documentation and visualization guides, visit our API Documentation

---

❓ Frequently Asked Questions

🤔 How does VCS differ from BLEU/ROUGE?

Unlike BLEU and ROUGE which rely on hard n-gram matching, VCS utilizes latent space matching by comparing embeddings at both global and local chunk levels. VCS also evaluates the chronological order of content chunks and combines these three dimensions to generate a comprehensive final score that better captures semantic similarity and narrative structure.

⚡ What's the minimum text length for VCS?

VCS works with any text length, but it's optimized for longer texts (100+ words) where narrative structure is important. For very short texts, simpler metrics might be more appropriate.

📏 What's the maximum text length for VCS?

There is no upper limit on text length for VCS. The framework is designed to handle texts of any size, from short paragraphs to extensive documents, making it suitable for large-scale narrative evaluation tasks.

🧠 Which embedding models work best?

We recommend checking the MTEB leaderboard for the latest SOTA models. As of 2024, nv-embed-v2 and similar transformer-based models provide excellent results.

🎯 How do I control the granularity of comparison?

Use the chunk_size parameter to control the granularity of text comparison. A smaller chunk size provides more fine-grained analysis, while a larger chunk size offers broader, more general comparisons. The default value is 1 for maximum granularity.

⏱️ How do I control the strictness of chronological matching?

Use the lct (Local Chronology Tolerance) parameter to control chronological matching strictness. A higher LCT value means more lenient chronological ordering, allowing for greater flexibility in narrative sequence evaluation. The default value is 0 for strict chronological matching.

🔗 Can I use different embedding functions for GAS and LAS?

Yes, you can specify different embedding functions for Global Alignment Score (GAS) and Local Alignment Score (LAS) using the embedding_fn_gas and embedding_fn_las parameters respectively. This allows you to optimize each component with models best suited for their specific evaluation tasks.

---

🏗️ Project Structure

vcs/
├── 📁 src/vcs/                  # Main package source code
│   ├── 📄 __init__.py           # Package initialization
│   ├── 📄 scorer.py             # Main VCS API entry point
│   ├── 📄 _config.py            # Configuration settings
│   ├── 📁 _metrics/             # Core VCS metrics implementations
│   │   ├── 📁 _gas/             # Global Alignment Score
│   │   ├── 📁 _las/             # Local Alignment Score  
│   │   ├── 📁 _nas/             # Narrative Alignment Score with components
│   │   │   └── 📁 _nas_components/  # Distance NAS, Line NAS, Regularize NAS
│   │   └── 📁 _vcs/             # Combined VCS computation
│   ├── 📁 _visualize_vcs/       # Comprehensive visualization suite
│   │   ├── 📁 _similarity_matrix/  # Similarity matrix visualizations
│   │   ├── 📁 _best_match/      # Best match analysis plots
│   │   ├── 📁 _distance_nas/    # Distance-based NAS visualizations
│   │   ├── 📁 _line_nas/        # Line-based NAS visualizations
│   │   ├── 📁 _mapping_windows/ # Context window visualizations
│   │   ├── 📁 _metrics_summary/ # Overall metrics summary plots
│   │   ├── 📁 _pdf_report/      # PDF report generation
│   │   ├── 📁 _text_chunks/     # Text chunk visualizations
│   │   ├── 📁 _window_regularizer/ # Window regularizer plots
│   │   ├── 📁 _las/             # LAS-specific visualizations
│   │   └── 📁 _config/          # Visualization configuration
│   ├── 📁 _segmenting/          # Text segmentation utilities
│   ├── 📁 _matching/            # Optimal text matching algorithms
│   ├── 📁 _mapping_windows/     # Context window management
│   └── 📁 _utils/               # Helper utilities
├── 📁 docs/                     # Documentation and interactive demos
│   ├── 📄 index.html            # Main documentation website
│   ├── 📁 pages/                # Documentation pages
│   │   ├── 📄 api.html          # API reference
│   │   ├── 📄 playground.html   # Interactive playground
│   │   └── 📄 example.html      # Usage examples
│   ├── 📁 widgets/              # Interactive visualization widgets
│   ├── 📁 sphinx/               # Sphinx documentation source
│   └── 📁 assets/               # Documentation assets (CSS, JS, videos)
├── 📁 .github/                  # GitHub configuration
│   ├── 📁 assets/               # README assets (images, gifs)
│   ├── 📁 scripts/              # GitHub automation scripts
│   └── 📁 workflows/            # CI/CD automation pipelines
│       ├── 📄 test.yml          # Continuous testing
│       ├── 📄 publish.yml       # Package publishing
│       └── 📄 deploy-docs.yml   # Documentation deployment
├── 📄 pyproject.toml           # Package configuration & dependencies
├── 📄 CONTRIBUTING.md          # Development contribution guide
├── 📄 DEPLOYMENT.md            # Release and deployment guide
├── 📄 CHANGELOG.md             # Version history and changes
├── 📄 MANIFEST.in              # Package manifest
├── 📄 tag_version.py           # Version tagging script
├── 📄 LICENSE                  # MIT license
└── 📄 README.md                # This documentation

---

🚀 Development & Contributing

We welcome contributions to VCS Metrics! Whether you're fixing bugs, adding features, or improving documentation, here's how to get started.

🛠️ Quick Development Setup

🖱️ Click to expand development setup

# 1. Clone and setup
git clone https://github.com/hdubey-debug/vcs.git
cd vcs
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# 2. Install development dependencies
pip install -e .[dev]

# 3. Create your feature branch
git checkout -b feature/your-feature-name

# 4. Make your changes
# Edit files in src/vcs/
# Add tests if needed
# Update docs if necessary

# 5. Run quality checks
black src/ && isort src/ && flake8 src/ && mypy src/

# 6. Commit with semantic format
git commit -m "minor: add new awesome feature"

# 7. Push and create PR
git push origin feature/your-feature-name

📋 Contribution Workflow

🔄 Development Process 1. Fork & Clone 2. Create Feature Branch 3. Make Changes 4. Write Tests 5. Submit PR 6. Code Review 7. Merge to Main ✅ Automated testing on every PR ✅ Fast feedback in ~2-3 minutes

📦 Release Process 1. Semantic Commit Messages 2. GitHub Release Creation 3. Automated Version Calculation 4. Package Building 5. TestPyPI Publishing 6. Production Release 🚀 Industry-standard CI/CD pipeline ⚡ Zero manual version management

💡 Semantic Commit Format

We use semantic commits for automatic version bumping:

Commit Type Version Bump Example minor: description New features 1.0.4 → 1.1.0 major: description Breaking changes 1.0.4 → 2.0.0 anything else Bug fixes (default) 1.0.4 → 1.0.5

🔧 Automated Testing & CI/CD

Our comprehensive CI/CD pipeline ensures code quality and reliability on every commit:

🚀 What Gets Tested ✅ Matrix Testing - Python 3.11 & 3.12 compatibility ✅ Package Validation - Import testing & API availability ✅ Integration Testing - Full getting-started example ✅ Code Quality - Flake8 linting & complexity checks ✅ Build Testing - Package build verification 🔄 Triggers: Every push and pull request to main

✅ Automated testing ensures every change is production-ready

📖 Detailed Guides

For comprehensive information about contributing and development:

🤝 Getting Help

🐛 Bug Reports Create GitHub Issue

💬 Questions GitHub Discussions

💡 Feature Requests Feature Request Issue

---

📚 Citation

If you use VCS Metrics in your research, please cite:

@software{vcs_metrics_2024,
  title = {VCS Metrics: Video Comprehension Score for Text Similarity Evaluation},
  author = {Dubey, Harsh and Ali, Mukhtiar and Mishra, Sugam and Pack, Chulwoo},
  year = {2024},
  institution = {South Dakota State University},
  url = {https://github.com/hdubey-debug/vcs},
  note = {Python package for narrative similarity evaluation}
}

---

🤖 CLIP-CC Ecosystem Integration

VCS is designed to work seamlessly with CLIP-CC Dataset for comprehensive video understanding evaluation.

🔄 Perfect Integration: VCS + CLIP-CC 🎥 CLIP-CC provides the data → Rich video dataset with human summaries 🔍 VCS provides the evaluation → Advanced narrative comprehension metrics 🏆 Together: Complete research pipeline → From data loading to evaluation

---

🏆 Meet Our Contributors

🌟 The VCS Team - Building the Future of Text Similarity

Harsh Dubey Lead Developer & Research Scientist South Dakota State University Commits Lines Files 2 49K 171 📋 Key Work: • VCS Algorithm Architecture • Visualization Engine • LAS, GAS, and NAS Metrics

🤖 Automated Contributors

Contributor	Role	Contributions	Badge
🤖 GitHub Actions	CI/CD Automation	Clean history setup

📊 Contribution Analytics

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🌟 Made with ❤️ by the VCS Team

Authors: Harsh Dubey, Mukhtiar Ali, Sugam Mishra, and Chulwoo Pack
Institution: South Dakota State University
Year: 2024

⭐ Star this repo • 🐛 Report Bug • 💡 Request Feature • 💬 Community Q&A