cloudflare-images-migrator 1.0.0

Enterprise-grade tool to migrate images to Cloudflare Images with persistent tracking and duplicate detection

Cloudflare Images Migration Tool

A powerful Python tool to automatically migrate all images in a codebase to Cloudflare Images, replacing local and external image references with optimized Cloudflare URLs.

Features

🔒 Enterprise Security & Quality

• Advanced Threat Detection: Multi-layer security validation with malware scanning
• File Signature Verification: Magic byte validation and MIME type checking
• Content Security Scanning: Deep analysis for malicious patterns and XSS threats
• URL Security Validation: HTTPS enforcement and domain reputation checking
• EXIF Data Sanitization: Removes potentially sensitive metadata
• Rate Limiting Protection: Prevents abuse and ensures compliance
• Premium Image Optimization: AI-powered quality enhancement and compression
• Responsive Variants: Automatic generation of multiple image sizes
• Enterprise Audit Logging: SOX, GDPR, HIPAA, PCI DSS compliance support

📊 Enterprise Tracking & Deduplication

• Persistent Duplicate Detection: Never re-uploads the same image across sessions
• Comprehensive Tracking Database: SQLite database with full metadata storage
• CSV Export Support: Export migration data for spreadsheet analysis
• Session Tracking: Unique migration IDs for audit trails
• Cross-Session Statistics: Performance metrics across multiple runs
• Multi-Level Duplicate Detection: File hash, URL hash, and path matching
• Cloudflare Library Integration: Checks existing images before uploading

🚀 Core Migration Features

• Enhanced Image Detection: Finds badges, GitHub assets, CDN images, and traditional formats
• Multi-Format Parsing: Supports HTML, CSS, JavaScript, TypeScript, Markdown, JSON, and more
• Batch Processing: Efficiently uploads multiple images with progress tracking
• Smart Deduplication: Avoids duplicate uploads using content hashing
• Zip Support: Works with both directories and zip files
• Safe Operation: Creates backups and supports dry-run mode
• Colored Logging: Beautiful console output with progress bars
• Flexible Configuration: Environment variables, config files, or command-line options

Installation

1. Clone or download this repository
2. Install dependencies:

   pip install -r requirements.txt
   

Prerequisites

Before using the tool, you need:

1. Cloudflare Account ID: Found in your Cloudflare dashboard
2. Cloudflare API Token: Create one with Images permissions

Getting Your Credentials

1. Account ID:

   • Go to Cloudflare Dashboard
   • Select your account
   • Find "Account ID" in the right sidebar

2. API Token:

   • Go to Cloudflare API Tokens
   • Click "Create Token"
   • Use "Custom token" template
   • Add permissions: Cloudflare Images:Edit
   • Choose your account in "Account Resources"

Usage

Basic Usage

python main.py /path/to/your/codebase --account-id YOUR_ACCOUNT_ID --api-token YOUR_API_TOKEN

Tracking & Statistics

# Show comprehensive statistics
python main.py /path/to/your/codebase --show-stats

# Export migration data to CSV
python main.py /path/to/your/codebase --export-csv ./migration-data.csv

# View statistics without running migration
python main.py --show-stats

Using Environment Variables

export CLOUDFLARE_ACCOUNT_ID="your_account_id"
export CLOUDFLARE_API_TOKEN="your_api_token"
python main.py /path/to/your/codebase

Using Configuration File

Create a config.yaml file:

account_id: "your_account_id"
api_token: "your_api_token"
dry_run: false
backup: true
batch_size: 10

Then run:

python main.py /path/to/your/codebase --config-file config.yaml

Command Line Options

Option	Description	Default
--account-id	Cloudflare Account ID	Required
--api-token	Cloudflare API Token	Required
--output-dir	Output directory for results	Auto-generated
--dry-run	Preview changes without making them	False
--backup/--no-backup	Create backup before modifying	True
--config-file	Path to configuration file	None
--verbose, -v	Enable verbose logging	False
--file-types	Comma-separated file extensions	Auto-detected
--exclude-dirs	Comma-separated directories to exclude	Common build dirs
--show-stats	Display comprehensive statistics	False
--export-csv	Export migration data to CSV file	None

Examples

Tracking & Statistics Examples

# View comprehensive statistics
python main.py --show-stats

# Export migration history to CSV
python main.py --export-csv ./my-migration-data.csv

# Run migration with statistics display
python main.py ./my-website --show-stats --verbose

Dry Run (Preview Changes)

python main.py ./my-website --dry-run --verbose

Process Specific File Types

python main.py ./project --file-types=".html,.css,.js"

Exclude Directories

python main.py ./app --exclude-dirs="node_modules,dist,build"

Process Zip File

python main.py ./website-backup.zip --output-dir ./migrated-website

Enterprise Security Examples

# Maximum security with audit report
python main.py ./project --security-level enterprise --generate-security-report

# Premium quality optimization
python main.py ./images --optimization-level aggressive --security-level enterprise

# Compliance-focused migration with tracking
python main.py ./app --security-level enterprise --backup --verbose --show-stats

Supported File Types

The tool automatically detects and processes these file types:

• Web Files: .html, .htm, .css, .js, .jsx, .ts, .tsx
• Documentation: .md, .markdown
• Configuration: .json, .xml, .yaml, .yml
• Stylesheets: .scss, .sass, .less

Enhanced Image Detection

The tool now detects a wide variety of image types:

Traditional Images

• PNG, JPEG, GIF, WebP, SVG, BMP, ICO with file extensions

Badge Services

• Shield.io badges (img.shields.io)
• GitHub status badges
• NPM badges
• Travis CI badges
• And many more badge services

GitHub Assets

• GitHub raw content (raw.githubusercontent.com)
• GitHub user content (user-images.githubusercontent.com)
• GitHub assets (github.com/*/assets/)

CDN Images

• Images served from CDNs without traditional extensions
• Image-like paths (containing 'icon', 'logo', 'banner', etc.)
• URLs with image-like query parameters

Supported Image Formats

• PNG, JPEG, GIF, WebP, SVG, BMP, ICO
• Maximum file size: 10MB
• Maximum dimensions: 12,000 pixels
• Maximum area: 100 megapixels

Configuration File Format

Create a config.yaml file with these options:

# Required
account_id: "your_cloudflare_account_id"
api_token: "your_cloudflare_api_token"

# Optional
dry_run: false
backup: true
batch_size: 10
retry_count: 3
timeout: 30

# File processing
file_types:
  - ".html"
  - ".css"
  - ".js"
  - ".jsx"
  - ".ts"
  - ".tsx"
  - ".md"
  - ".json"

exclude_dirs:
  - "node_modules"
  - ".git"
  - "dist"
  - "build"

How It Works

1. 📁 Preparation: Extracts zip files if needed, creates backups
2. 🔍 Enhanced Scanning: Recursively scans files for all types of image references
3. 🎯 Duplicate Detection: Checks against existing Cloudflare Images and local database
4. 🚀 Smart Upload: Uploads only new, unique images to Cloudflare Images
5. 📊 Tracking: Records all metadata in persistent SQLite database
6. ✏️ Replacement: Replaces old references with Cloudflare URLs
7. 📋 Comprehensive Report: Provides detailed migration statistics and history

Tracking & Statistics

Migration Database

The tool maintains a persistent SQLite database (cloudflare_images.db) that tracks:

• Every image processed across all sessions
• File metadata (size, dimensions, format, quality)
• Upload timestamps and session IDs
• Cloudflare URLs and variants
• File hashes for duplicate detection
• Performance metrics and statistics

Statistics Display

Use --show-stats to see:

• Session Statistics: Current migration metrics
• Total Statistics: All-time migration history
• Performance Metrics: Average upload times, success rates
• Duplicate Prevention: Images skipped due to existing uploads
• Size Optimization: Compression ratios and savings

CSV Export

Export your migration data with --export-csv filename.csv:

• Complete migration history
• File metadata and performance metrics
• Timestamps and session tracking
• Cloudflare URLs and variants
• Perfect for reporting and analysis

Output

Enhanced Output Example

15:30:45 - INFO - Starting migration of: ./my-website
15:30:45 - INFO - ✓ Successfully connected to Cloudflare Images API
15:30:45 - INFO - ✓ Loaded 1,250 existing Cloudflare Images for duplicate detection
15:30:45 - INFO - ✓ Connected to tracking database (3,847 images tracked)
15:30:45 - INFO - Created backup: ./my-website_backup
15:30:45 - INFO - Phase 1: Enhanced scanning for image references...
Scanning files: 100%|██████████| 150/150 [00:02<00:00, 75.0it/s]
15:30:47 - INFO - Found 156 image references (45 unique, 111 duplicates)
15:30:47 - INFO - Phase 2: Smart uploading with duplicate detection...
Uploading images: 100%|██████████| 23/23 [00:15<00:00, 1.5it/s]
15:31:02 - INFO - Upload complete: 23/23 images uploaded (100.0% success rate)
15:31:02 - INFO - Phase 3: Replacing image references in code...
Modifying files: 100%|██████████| 12/12 [00:01<00:00, 10.0it/s]

==================================================
Migration Summary
==================================================
Files processed: 150
Images found: 156 (45 unique, 111 duplicates)
Images uploaded: 23 (0 skipped as duplicates)
Files modified: 12
URLs replaced: 156
Average file size reduction: 45.2%
Time elapsed: 0:00:17

Session Statistics:
- New images uploaded: 23
- Duplicate images skipped: 0
- Security threats blocked: 0
- Quality optimizations applied: 23

Total Statistics (All Sessions):
- Total images tracked: 3,870
- Total migrations completed: 47
- Total file size saved: 2.3 GB
- Average success rate: 99.2%
==================================================

✓ Tracking data saved to: cloudflare_images.db
✓ Migration completed successfully!

Troubleshooting

Common Issues

1. "Failed to connect to Cloudflare Images API"

   • Check your account ID and API token
   • Ensure API token has Images permissions
   • Verify internet connection

2. "Database locked" errors

   • Ensure no other migration processes are running
   • Check database file permissions
   • Consider running with --verbose for more details

3. "File too large" errors

   • Cloudflare Images has a 10MB limit per image
   • Consider optimizing large images before migration

4. "No image references found"

   • Check if file types are supported
   • Use --verbose to see detailed scanning logs
   • Verify exclude directories aren't too broad

5. Duplicate detection issues

   • Check if images were previously uploaded with different settings
   • Use --show-stats to see duplicate detection statistics
   • Consider clearing the tracking database if needed

Advanced Troubleshooting

# View detailed statistics
python main.py --show-stats --verbose

# Export data for analysis
python main.py --export-csv debug-data.csv

# Check database status
python main.py --show-stats  # Shows database statistics

Getting Help

• Use --verbose flag for detailed logging
• Check the generated log file for full operation details
• Use --show-stats to understand tracking database status
• Export CSV data for external analysis

Environment Variables

You can set these environment variables instead of using command-line options:

• CLOUDFLARE_ACCOUNT_ID or CF_ACCOUNT_ID
• CLOUDFLARE_API_TOKEN or CF_API_TOKEN

Security Notes

• Never commit API tokens to version control
• Use environment variables or config files with proper permissions
• The tool only reads from source files and uploads to Cloudflare
• Backups are created locally for safety
• Tracking database contains metadata only, no sensitive data

Database Management

The tool creates a cloudflare_images.db SQLite database to track:

• All processed images and their metadata
• Migration sessions and timestamps
• Duplicate detection hashes
• Performance statistics

Database Location

• Default: ./cloudflare_images.db in the current directory
• Contains no sensitive data (API tokens, personal info)
• Safe to backup and share for analysis

Contributing

This tool is designed to be extensible:

• Add new file type parsers in src/parsers.py
• Extend configuration options in src/config.py
• Add new utility functions in src/utils.py
• Enhance tracking features in src/image_tracker.py

License

This project is provided as-is for educational and practical use.

---

Made with ❤️ for the Cloudflare Images community