spark-assets 1.0.0

SPARK - System for Processing & Asset Refinement Kit - Transform raw game assets into organized reference data

⚡ SPARK - System for Processing & Asset Refinement Kit

SPARK transforms raw game assets into organized, labeled reference data.

🚀 Quick Start

# 1. Install SPARK (downloads CLIP model automatically on first run)
pip install -e .

# 2. Run on MPKG + PNG assets (100% offline after first run)
spark pipeline --input game_resources/ --output processed/ --verbose

First Run: Auto-downloads CLIP model (~500MB)
Future Runs: 100% offline, no cloud access

Mercury Support (Optional)

# One-time: Configure npm with GitHub token
npm config set @aristocrattechnologiesinc:registry https://npm.pkg.github.com
npm config set //npm.pkg.github.com/:_authToken YOUR_GITHUB_TOKEN

# Then: merc auto-installs on first use
spark merc_pipeline --input mercury_resources/ --output processed/ --verbose

Note: SPARK auto-installs merc CLI if npm is configured. See DEMO.md for details.

🛠️ Requirements

Core Requirements

• Python 3.8+
• pip (Python package manager)
• Internet connection (first run only - to download CLIP model)

What Gets Installed Automatically

When you run pip install -e ., SPARK automatically:

1. ✅ Installs all Python dependencies (torch, transformers, opencv, etc.)
2. ✅ Downloads CLIP model (~500MB, one-time)
3. ✅ Configures offline mode for future runs

After first run: 100% offline processing, no cloud access required.

Optional (For Mercury Bundle Support)

• Node.js 14+ with npm
• merc CLI - Install globally:

  npm install -g @aristocrattechnologiesinc/merc
  

For Mercury setup, see DEMO.md.

✨ Features

• Single-command processing - Extract, classify, deduplicate in one step
• Composite image splitting - Automatically detects atlas/sprite-sheet images and splits them into individual assets
• Video / animation support - Extracts distinct scene frames from .ogv, .mp4, and other video formats
• Hybrid classification - MPKG metadata + OCR + CLIP semantic
• Smart deduplication - Perceptual hash-based (~96% reduction)
• Mercury support - DXT texture extraction via lonestar-tools
• Organized output - Clean folder structure by label
• One image = one asset - Every output file contains a single visual unit

📋 CLI Commands

Command	Purpose
spark pipeline	Process MPKG + PNG assets
spark merc_pipeline	Process Mercury bundles
spark build-reference	Build CLIP reference embeddings
spark flatten	Flatten hierarchical asset folders
spark extract	Parse .aep projects into position-verified golden assets

Run AEP extraction from input\AE_Files

# From repo root; writes output to output\ by default
spark extract --test-input input\AE_Files --verbose

📁 Output Structure

processed/
├── final_labeled_assets/
│   ├── spin_button/
│   ├── wild_symbol/
│   └── ...
└── pipeline_metadata.json

🔧 Python API

from spark import process_assets

results = process_assets(
    input_dir="game_resources/",
    output_dir="processed/"
)

print(f"Processed {results['unique_after_dedup']} unique assets")

📦 Project Structure

spark/
├── spark/                 # Main package
│   ├── cli.py             # CLI commands
│   ├── assets/            # Processing modules
│   │   ├── pipeline.py           # Core pipeline
│   │   ├── composite_splitter.py # Atlas/composite → single-asset splitting
│   │   └── merc_rip_bridge.py    # Mercury integration
│   └── detection/         # Detection utilities
├── examples/              # Usage examples
├── tests/                 # Unit tests
│   ├── test_core.py
│   ├── test_composite_splitter.py
│   └── test_pipeline.py
├── input/                 # Sample input data (for testing)
└── DEMO.md                # Getting started guide

Note: merc CLI tool is installed globally via npm, not bundled in the repository.

🛠️ Requirements

• Python 3.8+
• For Mercury: Node.js + lonestar-tools merc

📝 License

MIT License - see LICENSE

---

SPARK - Built for game asset processing ⚡