blackduck-heatmap-metrics 0.1.18

Black Duck scan heatmap metrics analyzer with interactive visualizations

Black Duck Heatmap Metrics Analyzer

A Python-based tool for analyzing Black Duck scan metrics from CSV files in zip archives. Generates interactive HTML dashboards with time series analysis, scan type evolution tracking, and year-based filtering.

Quick Start

# 1. Install the package
pip install -e .

# 2. Run analysis on your heatmap data
bdmetrics "path/to/heatmap-data.zip"

# 3. Open the generated report in your browser
# Output: report_YYYYMMDD_HHMMSS.html

For project group filtering:

bdmetrics "data.zip" --project-group "Demo" \
  --bd-url "https://your-server.com" \
  --bd-token "your-api-token"

Table of Contents

• Prerequisites
• Features
• Installation
• Usage
  • Command-Line Examples
  • Multi-CSV File Support
  • Choosing the Right Report Type
  • Performance Optimization
  • Command-Line Options Reference
  • Using Project Group Filter
• CSV Data Format
• Report Features
• How It Works
• Output Files
• Customization
• Browser Compatibility
• Troubleshooting

Prerequisites

Exporting Heatmap Data from Black Duck

Before using this tool, you need to export the heatmap data from your Black Duck server:

1. Access Black Duck Administration

   • Log in to your Black Duck server as an administrator
   • Navigate to System → Log Files

2. Download Heatmap Logs

   • In the Log Files section, locate the Heatmap logs
   • Select the time period you want to analyze
   • Click Download to export the data as a ZIP archive
   • The downloaded file will contain CSV files with scan metrics

3. Use the Downloaded ZIP

   • Save the downloaded ZIP file (e.g., heatmap-data.zip)
   • Use this ZIP file as input to the bdmetrics command

📖 Detailed Instructions: Black Duck Documentation - Downloading Log Files

Features

• 📦 Zip Archive Support: Reads CSV files directly from zip archives
• 🗂️ Multi-CSV Support: Handles multiple CSV files (different Black Duck instances) with aggregated view
• 📊 Interactive Charts: Plotly-powered visualizations with hover details
• 🎯 Black Duck Specific: Tailored for Black Duck scan heatmap data
• 📅 Multi-level Filtering: Filter by file, year, and project
• 🏢 Project Group Filtering: Filter by Black Duck project groups (includes nested sub-groups)
• 🔍 Scan Type Analysis: Track scan type distribution and evolution over time
• ✅ Success/Failure Metrics: Monitor scan success rates
• 📱 Responsive Design: Works on desktop and mobile devices
• 🚀 Performance Optimized: Configurable min-scans threshold and skip-detailed mode for large datasets
• 📑 Flexible Report Types: Choose between full interactive report or simplified static report
• 📈 Capacity Usage Monitoring: Track Scans Per Hour (SPH) against your hosted environment ceiling with over-capacity and warning alerts, per-hour project drill-down, and a time-series chart

Installation

From Source

1. Clone or download this repository
2. Install the package:

# Install in development mode (recommended for development)
pip install -e .

# Or install normally
pip install .

Using pip (once published to PyPI)

pip install blackduck-heatmap-metrics

Usage

After installation, you can use the bdmetrics command from anywhere:

bdmetrics path/to/your/heatmap-data.zip

Or use it as a Python module:

from blackduck_metrics import read_csv_from_zip, analyze_data, generate_chart_data, generate_html_report

# Read data
dataframes = read_csv_from_zip("path/to/heatmap-data.zip")

# Analyze
analysis = analyze_data(dataframes)
chart_data = generate_chart_data(dataframes)

# Generate report
generate_html_report(analysis, chart_data, "output_report.html")

Command-Line Examples

# Basic usage - generates report with default settings
bdmetrics "C:\Users\Downloads\heatmap-data.zip"

# Specify output folder
bdmetrics "path/to/data.zip" -o reports

# Set minimum scans threshold (default: 10)
# Only projects with 50+ scans will appear in trend charts
bdmetrics "path/to/data.zip" --min-scans 50

# Filter data from a specific year onwards (excludes older data)
bdmetrics "path/to/data.zip" --start-year 2020

# Filter data up to a specific year (excludes newer data)
bdmetrics "path/to/data.zip" --end-year 2024

# Filter to a specific year range
bdmetrics "path/to/data.zip" --start-year 2022 --end-year 2024

# Filter by Black Duck project group (requires Black Duck connection)
# Includes all projects in the specified group and all nested sub-groups
bdmetrics "path/to/data.zip" --project-group "Demo"

# Filter by project group with credentials passed as arguments
bdmetrics "path/to/data.zip" --project-group "Demo" --bd-url "https://your-server.com" --bd-token "your-token"

# Skip detailed year+project combinations for faster processing and smaller files
# Recommended for large datasets (reduces file size by ~36%)
bdmetrics "path/to/data.zip" --skip-detailed

# Generate simplified report without interactive filters
# Creates a smaller file that loads faster (no dynamic filtering)
bdmetrics "path/to/data.zip" --simple

# Combine options for optimal performance with large datasets
bdmetrics "path/to/data.zip" --min-scans 100 --skip-detailed --start-year 2020 -o reports

# Monitor capacity: flag hours where total SPH exceeds the hosted environment ceiling
bdmetrics "path/to/data.zip" --capacity-sph 1500

# Monitor capacity with a custom warning threshold (default: 80% of ceiling)
bdmetrics "path/to/data.zip" --capacity-sph 1500 --sph-warning-pct 70

# Combine capacity monitoring with other options
bdmetrics "path/to/data.zip" --capacity-sph 1500 --sph-warning-pct 70 --start-year 2026 --compress

# Show version
bdmetrics --version

# Show help
bdmetrics --help

Multi-CSV File Support

The tool automatically handles zip archives containing multiple CSV files, ideal for comparing different Black Duck SCA instances or environments.

Use Case Examples:

• Compare Production vs. Staging vs. Development Black Duck instances
• Analyze Regional instances (US, EU, APAC) in one report
• Track metrics across different Black Duck servers in your organization

How it works:

# Process a zip with multiple CSV files (different Black Duck instances)
bdmetrics "multi-instance-data.zip"

# The report will:
# 1. Show AGGREGATED statistics from all instances in the summary
# 2. Provide a FILE DROPDOWN to select specific instances
# 3. Dynamically update charts when you select an instance

Example zip structure:

multi-instance-data.zip
├── production-blackduck.csv      # 50,000 scans
├── staging-blackduck.csv          # 15,000 scans
└── development-blackduck.csv      # 8,000 scans

Report behavior:

• Default view (All Files): Shows aggregated 73,000 total scans
• File dropdown selection: Choose "production-blackduck.csv" → Shows only 50,000 scans
• Charts update: All visualizations filter to the selected instance
• Cross-instance comparison: Switch between files to compare metrics

Choosing the Right Report Type

Use Full Report (default) when you need:

• ✅ Interactive filtering by file, year, and project
• ✅ Ad-hoc exploration of specific projects
• ✅ Detailed drill-down analysis
• ✅ Dynamic chart updates based on selections
• 📊 Ideal for: Analysis, investigation, troubleshooting

Use Simple Report (--simple) when you need:

• ✅ Fastest page load times
• ✅ Smaller file size for sharing
• ✅ Static overview of all data
• ✅ No JavaScript complexity
• 📊 Ideal for: Reports, presentations, email attachments, archiving

Example decision matrix:

# Detailed analysis of specific teams → Full report
bdmetrics "data.zip" --project-group "Team A"

# Quick overview to share with management → Simple report
bdmetrics "data.zip" --simple

# Large dataset for detailed investigation → Full report with optimizations
bdmetrics "data.zip" --min-scans 100 --skip-detailed

# Large dataset for quick overview → Simple report with optimizations
bdmetrics "data.zip" --simple --min-scans 100 --skip-detailed --start-year 2024

Performance Optimization

For large datasets with thousands of projects:

• Use --min-scans to filter out low-activity projects from trend charts (default: 10)
• Use --max-projects to control the cap on projects included in trend charts (default: 1000, use 0 for no limit)
• Use --skip-detailed to skip year+project combination charts (saves ~36% file size)
• Use --start-year to exclude historical data before a specific year (e.g., --start-year 2020)
• Use --end-year to exclude data after a specific year (e.g., --end-year 2024); combine with --start-year for a year range
• Use --project-group to analyze only projects within a specific Black Duck project group
• Use --simple to generate a simplified report without interactive filters (smaller file size, faster loading)
• Example: Dataset with 37,706 projects → 7,261 projects (--min-scans 100) → 282 MB vs 456 MB baseline

Command-Line Options Reference

Option	Type	Default	Description
zip_file	Required	-	Path to zip file containing CSV heatmap data
-o, --output	Optional	. (current dir)	Output folder path (auto-generates filename)
--min-scans	Integer	10	Minimum scans for project to appear in trend charts
--max-projects	Integer	1000	Max projects in trend charts; use 0 for no limit (see performance note)
--skip-detailed	Flag	False	Skip year+project charts (reduces file size ~36%)
--simple	Flag	False	Generate simplified report without interactive filters
--start-year	Integer	None	Filter data from this year onwards (e.g., 2020)
--end-year	Integer	None	Filter data up to and including this year (e.g., 2024)
--project-group	String	None	Filter by Black Duck project group (includes all nested sub-groups)
--bd-url	String	$BD_URL	Black Duck server URL
--bd-token	String	$BD_API_TOKEN	Black Duck API token
--capacity-sph	Integer	120	Hosted environment SPH ceiling. Enables capacity monitoring with over-capacity/warning alerts in the report
--sph-warning-pct	Integer	80	Percentage of --capacity-sph that triggers a warning (default: 80). Pass 100 to track over-capacity hours only
--compress	Flag	False	gzip-compress HTML output as .html.gz; browsers open these natively
-v, --version	Flag	-	Show version and exit
-h, --help	Flag	-	Show help message and exit

Note: --bd-url and --bd-token are only required when using --project-group.

Using Project Group Filter

The --project-group option allows you to filter analysis to only include projects that are members of a specific Black Duck project group. This requires connecting to your Black Duck server.

Important: When you specify a project group, the tool will automatically include:

• ✅ All projects directly in the specified group
• ✅ All projects in any sub-project-groups (nested groups)
• ✅ All projects in sub-sub-project-groups (recursively traverses the entire hierarchy)

This means if you have a structure like:

Business Unit A
├── Team 1
│   ├── Project A
│   └── Project B
├── Team 2
│   ├── Subteam 2.1
│   │   └── Project C
│   └── Project D
└── Project E

Filtering by --project-group "Business Unit A" will include Projects A, B, C, D, and E.

Getting a Black Duck API Token:

1. Log in to your Black Duck server
2. Click on your username (top right) → My Access Tokens
3. Click Create New Token
4. Enter a name (e.g., "Heatmap Metrics Tool")
5. Set appropriate scope/permissions (read access to projects)
6. Click Create
7. Copy the token immediately (it won't be shown again)

Setup Black Duck Connection:

You can provide credentials in two ways:

Option 1: Environment Variables (Recommended for automation)

# On Windows (PowerShell)
$env:BD_URL = "https://your-blackduck-server.com"
$env:BD_API_TOKEN = "your-api-token-here"

# Or use username/password (API token is recommended)
$env:BD_URL = "https://your-blackduck-server.com"
$env:BD_USERNAME = "your-username"
$env:BD_PASSWORD = "your-password"

# On Linux/Mac
export BD_URL="https://your-blackduck-server.com"
export BD_API_TOKEN="your-api-token-here"

Option 2: Command-Line Arguments (Recommended for one-time use)

# Pass credentials directly via command-line
bdmetrics "path/to/data.zip" --project-group "Demo" \
  --bd-url "https://your-blackduck-server.com" \
  --bd-token "your-api-token-here"

Example Usage:

# Filter to only analyze projects in the "Business Unit A" group (using env vars)
bdmetrics "path/to/data.zip" --project-group "Business Unit A"

# Same, but with credentials as arguments
bdmetrics "path/to/data.zip" --project-group "Business Unit A" \
  --bd-url "https://your-server.com" --bd-token "your-token"

# Combine with other filters
bdmetrics "path/to/data.zip" --project-group "Demo" --start-year 2024 --min-scans 50

# Complete example: Project group + simplified report + optimizations
bdmetrics "heatmap-data.zip" \
  --project-group "Business Unit A" \
  --simple \
  --min-scans 100 \
  --skip-detailed \
  --start-year 2024 \
  --bd-url "https://blackduck.example.com" \
  --bd-token "your-token"
# Creates: report_YYYYMMDD_HHMMSS_Business_Unit_A.html (optimized simple report)

# Compress the output to reduce file size (browsers open .html.gz natively)
bdmetrics "heatmap-data.zip" --compress
# Creates: report_YYYYMMDD_HHMMSS.html.gz

# Compress combined with other options
bdmetrics "heatmap-data.zip" --project-group "Business Unit A" --simple --compress
# Creates: report_YYYYMMDD_HHMMSS_Business_Unit_A.html.gz

Running the bdmetrics command will:

1. Extract and read all CSV files from the zip archive
2. Analyze Black Duck scan metrics
3. Generate an interactive HTML report:
   • By default: Full report with all interactive filters (file, year, project): report_YYYYMMDD_HHMMSS.html
   • With --simple: Simplified report without filters (smaller, faster): report_YYYYMMDD_HHMMSS.html
   • With --project-group: Report includes project group name: report_YYYYMMDD_HHMMSS_<group-name>.html

CSV Data Format

The tool expects CSV files with the following columns:

• hour: Timestamp of the scan
• codeLocationId: Unique identifier for code location
• codeLocationName: Name of the code location
• versionName: Version being scanned
• projectName: Name of the project
• scanCount: Number of scans
• scanType: Type of scan (e.g., SIGNATURE, BINARY_ANALYSIS)
• totalScanSize: Total size of the scan
• maxScanSize: Maximum scan size
• state: Scan state (COMPLETED, FAILED, etc.)
• transitionReason: Reason for state transition

Multi-CSV File Support

If your zip archive contains multiple CSV files (e.g., from different Black Duck SCA instances):

• Aggregated View: The report shows combined statistics across all CSV files by default
• File Selector: In full reports, use the dropdown to view data from specific Black Duck instances
• Instance Comparison: Compare metrics across different Black Duck servers or environments

Example use case:

heatmap-data.zip
├── production-instance.csv     # Production Black Duck data
├── staging-instance.csv        # Staging Black Duck data
└── development-instance.csv    # Development Black Duck data

The report will:

• Display aggregated totals from all three instances in summary statistics
• Provide a file dropdown to filter charts by specific instance
• Enable cross-instance analysis and comparison

Report Features

The generated HTML dashboard includes:

Report Types

Each run generates one report based on your selection:

• Full Report (default): Interactive report with complete filtering capabilities (file, year, project)
  • Filename: report_YYYYMMDD_HHMMSS.html or report_YYYYMMDD_HHMMSS_<project-group>.html
• Simple Report (with --simple flag): Lightweight report without interactive filters
  • Filename: Same as full report
  • Faster loading, smaller file size, ideal for sharing

Report Type Comparison

Feature	Full Report	Simple Report (--simple)
File filter	✅ Dropdown (multi-instance support)	❌ Not available
Year filter	✅ Interactive dropdown	❌ Not available
Project search	✅ Type-ahead search	❌ Not available
Dynamic chart updates	✅ Real-time filtering	❌ Static data
Charts included	✅ All charts	✅ All charts
Summary statistics	✅ Aggregated + per-instance	✅ Aggregated only
File size	Larger	Smaller
Page load speed	Slower	Faster
Best for	Analysis & investigation	Sharing & reporting

Summary Section

Displays aggregated statistics across all CSV files in the zip archive:

• Total Files Processed: Number of CSV files analyzed (e.g., different Black Duck instances)
• Total Records: Aggregated scan records from all files
• Unique Projects: Combined count of distinct projects across all instances
• Total Scans: Aggregated total number of scans
• Successful Scans: Combined number of completed scans
• Failed Scans: Combined number of failed scans
• Success Rate: Overall percentage of successful scans
• Busiest / Quietest Hours: Clock hour (UTC) with the highest/lowest aggregate scan count

Each card displays a small ⓘ info icon that shows a descriptive tooltip on hover, explaining what the metric measures.

Note: When multiple CSV files are present, summary statistics represent the combined view of all Black Duck instances. Use the file filter to view instance-specific metrics.

Interactive Filters

Full Report includes:

• File Selector: Dropdown to filter by specific CSV file (Black Duck instance)
  • Shows "All Files" by default (aggregated view)
  • Lists each CSV file individually when multiple files are present
  • Dynamically updates all charts and statistics based on selection
  • Useful for comparing different Black Duck SCA instances or environments
• Year Selector: Filter all data and charts by year
• Project Search: Type-ahead project search with dynamic filtering
• Clear Filters: Reset all filters to show aggregated data across all files

Simple Report (generated with --simple flag):

• No interactive filters
• Static view of all data
• Smaller file size and faster page load

Charts and Visualizations

1. Scan Activity Over Time

   • Line chart showing number of scans over time (uses scanCount.sum() per hour bucket, not row count)
   • Total scan size trends
   • Filters by year, project, and file

2. Top Projects by Scan Count

   • Horizontal bar chart of top 20 projects
   • Updates based on filter selection

3. Scan Type Distribution

   • Pie chart showing breakdown of scan types
   • Updates based on year/project selection

4. Scan Type Evolution Over Time

   • Multi-line time series chart (uses scanCount.sum() per scan type per hour, not row count)
   • Interactive checkbox selection for scan types
   • Track how different scan types have evolved
   • Smart error messages when data unavailable (shows min-scans threshold)
   • Automatically updates when filters change

Smart Error Messages

The tool provides context-aware messages when data is unavailable:

• "No trend data for this project (project has less than X scans)" - when project doesn't meet min-scans threshold
• "Year+Project combination data not available" - when --skip-detailed flag was used

Capacity Usage – Scans Per Hour (SPH)

Enabled with --capacity-sph <N>. Adds a dedicated section to both report types:

• Peak SPH card — highest observed SPH value and its percentage of the configured ceiling
• Hours Over Capacity card — count of hours where total SPH ≥ ceiling
• Hours in Warning card — count of hours within the warning zone (≥ sph_warning_pct% of ceiling but below ceiling); hidden when --sph-warning-pct 100
• Scans Per Hour Over Time chart — Plotly time-series with colour-coded markers:
  • 🔴 Red — over capacity
  • 🟡 Amber — in warning zone
  • 🔵 Blue — normal
  • Dashed reference lines mark the capacity ceiling and warning threshold
• Over Capacity & Warning Hours drill-down table — each flagged hour shows its SPH, % of capacity, status badge (EXCEEDED / WARNING), Snippet % (percentage of that hour's scans that are SNIPPET type, colour-coded: red ≥ 50%, amber ≥ 25%, green < 25%), and the top contributing projects sorted by scan count descending

SPH is calculated as scanCount.sum() per hour bucket. The SPH section is always visible in reports; pass --capacity-sph <N> to override the default ceiling of 120.

Black Duck Overview

• Scan type breakdown with counts
• State distribution
• Filterable statistics

Requirements

• Python 3.7+
• pandas >= 3.0.1
• jinja2 >= 3.1.6
• plotly >= 6.6.0
• tqdm >= 4.67.3
• blackduck >= 1.1.3
• requests >= 2.32.5

Project Structure

blackduck_heatmap_metrics/
├── blackduck_metrics/
│   ├── __init__.py              # Package initialization
│   ├── analyzer.py              # Core data analysis and report generation
│   ├── blackduck_connector.py   # Black Duck SCA connection handler
│   ├── cli.py                   # Command-line interface
│   └── templates/
│       ├── template.html        # Full report template (interactive filters)
│       └── template_simple.html # Simple report template (no filters)
├── setup.py                     # Package installation script
├── pyproject.toml               # Project metadata
├── requirements.txt             # Python dependencies
├── MANIFEST.in                  # Package manifest
└── README.md                    # This file

How It Works

1. Data Extraction: Reads all CSV files from zip archive using pandas
   • Supports single or multiple CSV files (e.g., different Black Duck instances)
   • Each CSV file is processed and tracked separately
2. Project Filtering (optional): Connects to Black Duck to filter projects by project group
3. Time-based Analysis: Parses timestamps and groups data by year and project
4. Aggregation: Calculates statistics per file, year, project, and year+project combinations
   • Generates both aggregated (all files) and individual file statistics
   • Enables cross-instance comparison when multiple CSV files are present
5. Chart Generation: Prepares optimized data structures for Plotly visualizations
   • Applies min-scans threshold to filter low-activity projects
   • Optionally skips year+project combinations for performance
   • Reduces data sampling for large datasets (time series: 200 points, scan type evolution: 100 points)
6. Template Rendering: Jinja2 combines data with selected template (full or simple)
7. Output: Generates a timestamped HTML file with embedded charts and interactive file selector (when multiple CSVs)

Customization

Template Styling

Edit templates in blackduck_metrics/templates/:

• template.html - Full report with interactive filters
• template_simple.html - Simple report without filters
• Customize: Color scheme (blue gradient), chart types, layouts, summary cards, fonts

Data Analysis

Modify blackduck_metrics/analyzer.py to:

• Add new aggregations
• Include additional metrics
• Change chart data calculations
• Adjust min-scans thresholds
• Modify data sampling rates
• Adjust filtering logic

Output Files

Each run generates one report with a timestamp-based filename:

Default Filename Format

• Basic: report_YYYYMMDD_HHMMSS.html
• With project group: report_YYYYMMDD_HHMMSS_<sanitized-group-name>.html
• With -o folder: Output to specified folder with timestamped filename

Examples

# Default full report
bdmetrics "data.zip"
# Output: report_20260216_143015.html

# Simple report
bdmetrics "data.zip" --simple
# Output: report_20260216_143015.html

# With project group
bdmetrics "data.zip" --project-group "Business Unit A"
# Output: report_20260216_143015_Business_Unit_A.html

# With project group and simple
bdmetrics "data.zip" --project-group "Demo" --simple
# Output: report_20260216_143015_Demo.html

# Custom output folder
bdmetrics "data.zip" -o reports
# Output: reports/report_<timestamp>.html

Report Characteristics

All generated reports are:

• Standalone HTML files (no external dependencies except Plotly CDN)
• Self-contained with embedded data and charts
• Shareable - can be opened directly in any modern browser
• Single file per execution (either full or simple, based on flags)

Browser Compatibility

The generated reports work in all modern browsers:

• Chrome/Edge (recommended)
• Firefox
• Safari
• Opera

Requires JavaScript enabled for interactive features.

Troubleshooting

No charts showing

• Check browser console (F12) for JavaScript errors
• Ensure Plotly CDN is accessible
• Verify CSV data has the expected columns

Charts show "No trend data for this project (project has less than X scans)"

• This is normal for projects with few scans
• Adjust --min-scans threshold if needed (default: 10)
• In full reports, click "Clear Filters" to see all data
• Simple reports show all data without filtering

Charts not updating after filter selection

• Ensure JavaScript is enabled
• Try refreshing the page
• Check browser console for errors

"Year+Project combination data not available" message

• Report was generated with --skip-detailed flag
• Regenerate without this flag for full year+project filtering
• This is normal for optimized reports

Report file too large

• Use --simple to generate a report without interactive filters (significantly smaller)
• Use --min-scans 50 or higher to reduce projects in charts
• Use --skip-detailed to skip year+project combinations (~36% size reduction)
• Use --start-year to exclude historical data
• Example: 456 MB → 282 MB with --min-scans 100 --skip-detailed

Filters not available or working

• If using --simple flag, filters are not included by design (use default mode for filters)
• In full reports, ensure JavaScript is enabled
• Ensure hour column contains valid timestamps
• Check that data spans multiple years for year filtering

Charts show "No data available"

• Verify CSV files contain the required columns
• Check for empty or malformed data
• Ensure project has sufficient scans (check min-scans threshold)

License

MIT License