multimodal-analyzer 0.1.0

AI-powered media analysis tool for images, audio, and video using multiple LLM providers

Multimodal Analyzer CLI

AI-powered media analysis tool using multiple LLM providers through LiteLLM. Analyze images, audio, and video files with customizable prompts and output formats.

Features

• Multi-model Support: Use Gemini, OpenAI, Claude, and more through LiteLLM
• Image, Audio & Video Analysis: Single files or batch process entire directories
• Streaming JSON Input: Multi-turn conversations via stdin JSONL for interactive analysis
• Hybrid File Input: Specify files by directory path OR explicit file lists from multiple locations
• Automatic Image Preprocessing: Images > 500KB are automatically converted to JPEG for optimal processing
• Concurrent Processing: Configurable concurrency with progress tracking
• Multiple Output Formats: JSON, Markdown, Text, and streaming JSON export
• Custom Prompts: Flexible analysis with custom or predefined prompts

Installation

Prerequisites

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

Install from Source (Recommended for Users)

To install and use the tool system-wide:

# Clone the repository
git clone https://github.com/sarath-menon/multimodal-analyzer.git
cd multimodal-analyzer

# Install globally with uv (specify Python 3.12.8)
uv tool install --python 3.12.8 .

After installation, the multimodal-analyzer command will be available system-wide.

Development Setup

For development and contributing:

# Clone the repository
git clone https://github.com/sarath-menon/multimodal-analyzer.git
cd multimodal-analyzer

# Install dependencies
uv sync

# Run commands with uv run prefix
uv run multimodal-analyzer --help

Verify Installation

# Check if installation worked
multimodal-analyzer --version

# Or for development setup
uv run multimodal-analyzer --version

Reinstalling

To update or reinstall:

cd multimodal-analyzer
git pull
uv tool install --python 3.12.8 --force-reinstall .

Hybrid File Input Support

The Multimodal Analyzer CLI supports two flexible input modes:

Directory Path Mode (--path)

Use --path to analyze files from directories or single files:

# Single file
multimodal-analyzer --type image --model gemini/gemini-2.5-flash --path photo.jpg

# Directory (all supported files)
multimodal-analyzer --type image --model gemini/gemini-2.5-flash --path ./photos/

# Recursive directory scan
multimodal-analyzer --type image --model gemini/gemini-2.5-flash --path ./dataset/ --recursive

Explicit File List Mode (--files)

Use --files to specify exact files from multiple locations:

# Multiple files from different directories
multimodal-analyzer --type image --model gemini/gemini-2.5-flash \
  --files /home/user/photo1.jpg \
  --files /work/project/chart.png \
  --files ./local/screenshot.jpg

# Audio files from various locations
multimodal-analyzer --type audio --model gemini/gemini-2.5-flash \
  --files recording1.mp3 \
  --files /meetings/call.wav \
  --audio-mode transcript

When to Use Each Mode

• Use --path for processing all files in a directory or subdirectories
• Use --files for selective processing of specific files from multiple locations
• Cannot use both --path and --files simultaneously (mutually exclusive)

Image Analysis Usage

Basic Image Commands

# Analyze single image
multimodal-analyzer --type image --model gemini/gemini-2.5-flash --path photo.jpg

# Batch process directory
multimodal-analyzer --type image --model azure/gpt-4.1-mini --path ./photos/ --output markdown

# Development installation (prefix with uv run)
uv run multimodal-analyzer --type image --model gemini/gemini-2.5-flash --path photo.jpg

Advanced Image Analysis

# Custom prompt with word count
multimodal-analyzer --type image --model claude-3-sonnet-20240229 --path chart.jpg \
  --prompt "Analyze this chart focusing on data insights" --word-count 300

# Recursive batch processing
multimodal-analyzer --type image --model gpt-4o-mini --path ./dataset/ \
  --recursive --concurrency 5 --output json --output-file results.json

# Analyze specific images from multiple directories
multimodal-analyzer --type image --model gpt-4o-mini \
  --files ./screenshots/chart1.png \
  --files ./photos/diagram.jpg \
  --files /tmp/analysis_image.png \
  --prompt "Compare these visuals" --word-count 200

Audio Analysis Usage

Basic Audio Commands

# Transcribe audio
multimodal-analyzer --type audio --model whisper-1 --path audio.mp3 --audio-mode transcript

# Analyze audio content
multimodal-analyzer --type audio --model gpt-4o-mini --path podcast.wav --audio-mode description

Advanced Audio Processing

# Batch transcription
multimodal-analyzer --type audio --model whisper-1 --path ./audio/ \
  --audio-mode transcript --output text --output-file transcripts.txt

# Content analysis with custom prompts
multimodal-analyzer --type audio --model gpt-4o-mini --path podcast.wav \
  --audio-mode description --prompt "Summarize key insights" --word-count 200

# Transcribe specific audio files from different locations
multimodal-analyzer --type audio --model whisper-1 \
  --files ./meetings/standup.mp3 \
  --files ./interviews/candidate1.wav \
  --files /recordings/conference_call.m4a \
  --audio-mode transcript --output markdown --output-file transcripts.md

Video Analysis Usage

Note: Video analysis is currently restricted to Gemini models only due to native multimodal video support.

Basic Video Commands

# Analyze video content (Gemini only)
multimodal-analyzer --type video --model gemini/gemini-2.5-flash --path video.mp4 --video-mode description

Advanced Video Analysis

# Single video analysis
multimodal-analyzer --type video --model gemini/gemini-2.5-flash --path presentation.mp4 \
  --video-mode description --word-count 150

# Batch video processing with custom prompts
multimodal-analyzer --type video --model gemini/gemini-2.5-flash --path ./videos/ \
  --video-mode description --prompt "Describe the visual content and any audio" \
  --recursive --output markdown --output-file video_analysis.md

# Video analysis with detailed output
multimodal-analyzer --type video --model gemini/gemini-2.5-flash --path tutorial.mp4 \
  --video-mode description --verbose --word-count 200

# Analyze specific videos from multiple projects
multimodal-analyzer --type video --model gemini/gemini-2.5-flash \
  --files ./project1/demo.mp4 \
  --files ./project2/presentation.avi \
  --files /shared/training_video.mov \
  --video-mode description --prompt "Focus on key features demonstrated" \
  --word-count 300 --output json --output-file video_summaries.json

Streaming JSON Input

Enable interactive, multi-turn conversations by providing JSON messages via stdin. Each line of input is a complete JSON message in JSONL format, allowing continuous analysis without re-launching the CLI.

Basic Streaming Usage

# Start streaming mode for image analysis
echo '{"type":"user","message":{"role":"user","content":[{"type":"text","text":"Explain this image"},{"type":"image_url","image_url":{"url":"data:image/jpeg;base64,..."}}]}}' | \
  multimodal-analyzer --type image --model gemini/gemini-2.5-flash -p . --input-format stream-json --output stream-json

Streaming Requirements

• Required flags: --input-format stream-json + --output stream-json + -p
• Incompatible with: --files, --output-file, batch processing options
• Current support: Image analysis only (audio/video coming soon)

Message Format

Input Message Structure:

{
  "type": "user",
  "message": {
    "role": "user", 
    "content": [
      {"type": "text", "text": "Describe this image"},
      {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,<base64-data>"}}
    ]
  }
}

Response Format:

{
  "type": "assistant",
  "message": {
    "role": "assistant",
    "content": "AI analysis response..."
  },
  "metadata": {
    "success": true,
    "model": "gemini/gemini-2.5-flash"
  }
}

Interactive Usage Examples

# Multi-turn conversation with different images
cat input.jsonl | multimodal-analyzer --type image --model gemini/gemini-2.5-flash -p . --input-format stream-json --output stream-json

# Real-time processing with custom prompts
echo '{"type":"user","message":{"role":"user","content":[{"type":"text","text":"What colors do you see?"},{"type":"image_url","image_url":{"url":"data:image/jpeg;base64,..."}}]}}' | \
  multimodal-analyzer --type image --model gpt-4o-mini -p . --input-format stream-json --output stream-json --word-count 50

Models

Supports any model available through LiteLLM.

Command Line Options

Options:
  -t, --type [image|audio|video]  Analysis type: image, audio, or video (required)
  -m, --model TEXT                LiteLLM model (required)
  -p, --path PATH                 Media file or directory path (mutually exclusive with --files)
  -f, --files TEXT                Explicit list of media files to analyze (mutually exclusive with --path)
  --audio-mode [transcript|description] Audio analysis mode (required for audio type)
  --video-mode [description]      Video analysis mode (required for video type)
  -w, --word-count INTEGER        Target description word count [default: 100]
  --prompt TEXT                   Custom analysis prompt
  -o, --output [json|markdown|text|stream-json] Output format [default: json]
  --input-format [stream-json]    Input format for streaming mode (requires --output stream-json and -p)
  --output-file TEXT              Save results to file
  -r, --recursive                 Process directories recursively (only with --path)
  -c, --concurrency INTEGER       Concurrent requests [default: 3]
  --log-level [DEBUG|INFO|WARNING|ERROR] Logging level [default: INFO]
  -v, --verbose                   Show detailed output including model info
  --version                       Show version and exit
  --help                          Show help and exit

Standard Mode: You must specify either --path OR --files, but not both. Use --path for directory processing and --files for explicit file lists.

Streaming Mode: Requires --input-format stream-json, --output stream-json, and -p. Cannot be used with --files or --output-file.

Output Schema

JSON Output Format (Batch Mode)

Results are returned as an array of objects, one per analyzed file:

[
  {
    "image_path": "path/to/image.jpg",
    "analysis": "AI-generated analysis text...",
    "success": true
  }
]

Streaming JSON Output Format

Each response is a single JSON object written immediately to stdout:

{
  "type": "assistant",
  "message": {
    "role": "assistant",
    "content": "AI-generated analysis text..."
  },
  "metadata": {
    "success": true,
    "model": "gemini/gemini-2.5-flash"
  }
}

Error Handling

Batch Mode - Failed analyses include error details:

[
  {
    "image_path": "path/to/image.jpg",
    "analysis": null,
    "success": false,
    "error": "Error message"
  }
]

Streaming Mode - Errors are returned as immediate JSON responses:

{
  "type": "assistant",
  "message": {
    "role": "assistant",
    "content": ""
  },
  "metadata": {
    "success": false,
    "model": "gemini/gemini-2.5-flash",
    "error": "Error message"
  }
}

Authentication

Google OAuth

For Gemini models, you can use OAuth authentication instead of API keys:

# Authenticate with Google OAuth
multimodal-analyzer auth login

# Check authentication status
multimodal-analyzer auth status

# Clear stored credentials
multimodal-analyzer auth logout

Development

Running Tests

Important: All tests require valid API keys and fail immediately if missing.

Run tests

uv run pytest uv run pytest --cov # with coverage

### Development Philosophy

- **Fail Fast**: Raise exceptions immediately rather than graceful degradation
- **Explicit Testing**: Never skip tests due to missing API keys - fail instead
- **Clear Errors**: All error conditions must raise descriptive exceptions

## Examples

See the `examples/` directory for:

- **`sample_usage.py`**: Comprehensive programmatic usage examples
- Different analysis scenarios and configuration options
- Batch processing examples with progress tracking

Run the examples:

```bash
uv run python examples/sample_usage.py

Debug Mode

Enable detailed logging for troubleshooting:

multimodal-analyzer --type image --model gemini/gemini-2.5-flash --path image.jpg --log-level DEBUG

Publishing

To publish a new version to PyPI:

1. Update version: Edit version in pyproject.toml
2. Build package: uv build --no-sources
3. Publish: uv publish --token YOUR_PYPI_TOKEN

For testing, publish to TestPyPI first:

uv publish --token YOUR_TEST_PYPI_TOKEN --index testpypi