google-photo-meta-fixer 1.0.0

CLI tool to sync and clean Google Photos backups on Google Drive

📸 Google Photos → Google Drive Backup & Cleanup Tool

A unified python-based toolkit and React Native companion app to process, sync, and fix timestamps of Google Photos backups on Google Drive using rclone. This project is designed to identify and organize original-quality files that consume storage quota and manage Google Photos metadata.

---

⚙️ Prerequisites

To use this toolkit, you must have rclone installed and configured on your system:

1. Install rclone: Follow the rclone installation instructions.
2. Configure your Google Drive remote: Set up your Google Drive connection by running:

   rclone config
   

   Follow the step-by-step prompts to create a new remote (e.g. gdrive:). Refer to the official rclone config command guide for more details.
3. Verify configuration: Make sure you can list your drive contents:

   rclone lsd gdrive:
   

To run the React Native Android companion app, you will also need:

• Node.js (v18 or higher) and npm installed.
• Expo Go app installed on your Android device (or an Android emulator configured via Android Studio).

---

🚀 High-Level End-User Guide

Follow this standard sequence to backup, audit, and clean up your Google Photos storage:

1. Install Browser Userscript:

   • Install the Tampermonkey browser extension.
   • Install the google_photos_toolkit.user.js userscript (located in the root of this repo).
   • Go to photos.google.com, filter by "Consuming" space / "Original" quality, and click Export Metadata to download a CSV file. Place it in data/csv/.

2. Google Drive Sync (Cloud-to-Cloud):

   • Run the python CLI or the companion Android app to compare your exported CSV metadata against your actual Google Drive.
   • Automatically copy missing files cloud-to-cloud to your backup directory.

3. Verify & Fix Timestamps:

   • Run the metadata touch tools to align Drive modification timestamps with your photo's local EXIF/CSV datetimes using rclone touch without re-uploading payloads.

4. Pixel Re-upload & Deletion Workflow (Free Storage Sync):

   • To safely clean up cloud storage and re-upload original files via Pixel phone, see the Pixel Re-upload Guide.

---

🏗️ Project Architecture & Code Structure

.
├── cleaner.py                 # Main CLI entrypoint (argparse interface)
├── google_photos_toolkit.user.js # Browser userscript for metadata export & Drive sync
├── src/                       # Core python modules
│   ├── sync.py                # Cloud-to-cloud file backup copying logic
│   ├── metadata.py            # Timestamp comparison and Drive fixing via 'rclone touch'
│   └── process_backup.py      # Parses CSV & updates local file modification times
├── android-app/               # React Native/Expo Android companion application
├── data/                      # Contains inputs/caches (Ignored by Git except templates/configs)
│   ├── csv/                   # Input metadata exports from Google Photos Toolkit (GPTK)
│   └── json/                  # Drive file listings (drive_index_photo_backUp.json, etc.)
└── logs/                      # Executables and commands logs (rclone commands lists)

---

📦 Installation & Packaging

You can run the tool directly as a script, install it locally as a system-wide CLI command, or compile it into a single standalone executable binary.

1. Run Directly as a Script

Run directly from the repository root:

python3 cleaner.py [command] [options]

2. Install as a Local CLI Command

You can install the tool as a system-wide CLI package (mapped to the command gp-cleaner):

# Install in editable mode
pip install -e .

# If using an externally managed environment (e.g. Homebrew on macOS)
pip install -e . --break-system-packages --user

Once installed, you can invoke the tool from any directory:

gp-cleaner [command] [options]

3. Build a Standalone macOS Executable Binary

To package the app into a single compiled binary that doesn't depend on system Python path settings:

1. Install PyInstaller:

   pip install pyinstaller --break-system-packages --user
   

2. Compile the binary:

   pyinstaller --onefile cleaner.py --name gp-cleaner
   

3. Run the standalone binary from the dist/ directory:

   ./dist/gp-cleaner [command] [options]
   

---

🔧 CLI Entry Point

All operations are run via cleaner.py (or gp-cleaner if installed) using positional commands:

1. File Synchronization (sync)

Handles synchronizing files inside Google Drive (cloud-to-cloud) or uploading missing local folders.

# Sync original-quality backup photos into year-wise folders
gp-cleaner sync backup --remote gdrive:

# Compare space-consuming CSV metadata against Google Drive index (cloud-to-cloud sync)
# This finds files present on Drive but not in photos_backUp, and copies them to the backup directory
gp-cleaner sync consuming --csv "o consuming album metadata.csv" --remote gdrive:

# Compare a local folder (e.g. GPH OP) against Drive index and upload missing files to a destination folder on Drive
# (Prevents duplicate files and folder structures)
gp-cleaner sync upload-local --dir "data/GPH OP/ALL_PHOTOS" --dest "O Consuming" --remote gdrive:

2. Metadata & Timestamps (metadata)

Identifies, verifies, and fixes timestamp discrepancies between local files, Google Drive, and CSV database reports.

# Compare local and Drive indices, and execute 'rclone touch' directly on Drive to fix mismatched times
gp-cleaner metadata fix-drive --remote gdrive:

# Fix local photo modification timestamps
gp-cleaner metadata fix-local

# Verify that local photo timestamps (EXIF & modify date) match dates in a Google Photos CSV file
gp-cleaner metadata verify-csv --csv "o consuming album metadata.csv" --dir "data/GPH OP/ALL_PHOTOS"

3. CSV Processing & Local Organizing (process)

Processes local files against GPTK metadata CSVs to extract correct timestamps and write them to the local files system (os.utime).

# Process and update timestamps for local backup photos
gp-cleaner process backup

4. Google Takeout Metadata Merger (process takeout)

Recursively walks a Google Takeout directory to find companion JSON files, extract timestamps, descriptions, and GPS tags, and embed them into the photos/videos.

# Process and merge Google Takeout directory metadata
gp-cleaner process takeout --dir <takeout_dir>

# Verify Google Takeout metadata updates
gp-cleaner metadata verify-takeout

---

📱 React Native Android Companion App

The project includes an Expo-based Android companion application located under the android-app/ directory. This app allows you to:

1. View and index local photo directories.
2. Connect to the Google Photos API to audit large storage-consuming files.
3. Track and verify Google Takeout processing logs directly on your mobile device.

Running the App:

1. Install Dependencies:

   cd android-app
   npm install
   

2. Start the Development Server:

   npx expo start
   

3. Open on Device:

   • Scan the QR code displayed in your terminal using the Expo Go app on your Android device.
   • Alternatively, press a in your terminal to open it in a running Android emulator.

---

🧠 Core System Design & Workflow

1. Data Source (Google Photos Toolkit)

Because the official Google Photos API strips GPS/EXIF metadata and hides the takesUpSpace attribute, this toolkit relies on the Google Photos Toolkit (GPTK) userscript (google_photos_toolkit.user.js in the root).

1. Run the userscript in Tampermonkey on photos.google.com.
2. Filter for "Consuming" space and "Original" quality.
3. Click "Export Metadata" to get metadata.csv.
4. Save the files under data/csv/.

2. Google Drive Indexing

We index files on Google Drive using rclone to avoid querying the API repeatedly:

rclone lsjson -R "gdrive:photos_backUp" > data/json/drive_index_photo_backUp.json

3. Local Comparison & Cloud Execution

1. The script loads the local files (data/photos/photos_backUp) and indexes their sizes and modification times.
2. It compares them to the cached drive_index_photo_backUp.json.
3. If timestamps mismatch (e.g., due to timezone shifts or upload bugs), the metadata fix-drive command uses rclone touch to adjust the remote file's modified time without re-uploading the file payload.

4. Google Takeout Metadata Merger & Matching Logic

For a complete guide of the algorithms, code snippets, and design choices behind this feature, see LEARNING.md. Below is a summary of the core concepts:

• Fuzzy Filename Matching: Resolves Takeout quirks such as name truncations, supplemental suffixes (e.g. .supplemental-metadata.json, .supp.json), double dots (..json), and duplicate bracket shifts (e.g. photo.jpg(1).json matching photo(1).jpg).
• Batch Exiftool Operations: Minimizes process startup overhead by writing all modifications to a single temporary JSON and importing it in one command using exiftool -json=temp.json -@ files.txt.
• Filesystem Date Correction: Updates the filesystem mtime using Python's os.utime() to match the JSON's true epoch timestamp (seconds since epoch) so that Finder and File Explorer sort photos chronologically.

---

⚠️ Important Rules for AI Developers

• No Standalone Scripts: Do not create new top-level run_*.sh or *.py scripts in the root or scripts/. Always implement logic as functions inside the src/ modules and register them as subcommands in cleaner.py.
• Paths: Keep paths relative to the workspace. Local raw photo directories are typically ignored by git (e.g., data/photos/).
• Parallelism: When generating mass rclone commands, write the commands to logs/ and execute them concurrently via run_rclone_commands in src/sync.py to optimize speed.