fedops-vlm-framework 0.1.0

Plugin-based federated learning framework for Vision Language Models (VLMs)

FedOps VLM Framework

This is a separate framework-oriented folder (not a single experiment run).

Goal

Provide a reusable Flower-based VLM framework with:

• model plugins (onevision, phiva)
• dataset plugins (vqav2, vqa_rad, and future multimodal sets)
• deployment plugins (mlc_compatible, onevision_research)
• runtime backends (mlc, llama_cpp) from one shared export request schema

Current status

• Framework CLI + plugin registry + runtime planning is active.
• Existing project tracks remain in:
  • /home/ccl/Desktop/akeel_folder/MMFL_Flower/fedops-vlm/projects/onevision-research
  • /home/ccl/Desktop/akeel_folder/MMFL_Flower/fedops-vlm/projects/mlc-compatible

Quick start

cd /home/ccl/Desktop/akeel_folder/MMFL_Flower/fedops-vlm/fedops-vlm-framework
source /home/ccl/Desktop/akeel_folder/MMFL_Flower/akeel_research_env/bin/activate
pip install -e .
python -m fedops_vlm_framework.cli --help

List runtime backends:

python -m fedops_vlm_framework.cli --list-runtimes

Generate a full end-to-end setup bundle (recommended first step):

python -m fedops_vlm_framework.cli --setup-e2e --track mlc-compatible

Or for OneVision research:

python -m fedops_vlm_framework.cli --setup-e2e --track onevision-research

This creates a timestamped folder under fedops-vlm/exports/ containing:

• manifest.json
• README_NEXT_STEPS.txt
• scripts/00_verify_env.sh
• scripts/01_train_fl.sh
• scripts/02_export_merged.sh
• scripts/03_generate_runtime_plans.sh
• scripts/04_run_mlc_pipeline.sh
• scripts/05_collect_a24_metrics.sh

Generate an MLC export plan (runtime-agnostic interface -> runtime-specific commands):

python -m fedops_vlm_framework.cli \
  --export-plan-runtime mlc \
  --base-model nota-ai/phiva-4b-hf \
  --adapter-dir /path/to/results/<run>/adapter_10 \
  --output-dir /path/to/exports/phiva_a24 \
  --device-profile samsung_a24 \
  --quantization q4f16_0 \
  --context-window-size 768 \
  --image-size 224

Generate the same plan from a JSON config (recommended for repeatability):

python -m fedops_vlm_framework.cli \
  --plan-config examples/plan_config.phiva.mlc.json \
  --save-plan /tmp/plan_mlc.json

Generate a llama.cpp export plan:

python -m fedops_vlm_framework.cli \
  --export-plan-runtime llama_cpp \
  --base-model nota-ai/phiva-4b-hf \
  --adapter-dir /path/to/results/<run>/adapter_10 \
  --output-dir /path/to/exports/phiva_a24 \
  --runtime-option gguf_quant=Q4_K_M

Or config-driven:

python -m fedops_vlm_framework.cli \
  --plan-config examples/plan_config.phiva.llama_cpp.json \
  --save-plan /tmp/plan_llama_cpp.json

Adding new models/datasets/runtimes

No large refactor is needed for normal extension:

1. Add one plugin file under fedops_vlm_framework/plugins/{models|datasets|runtime}.
2. Register it in core/registry.py (models/datasets/deploy) or core/runtime_registry.py (runtime).
3. Use CLI planning with either flags or --plan-config.

Suggested next step

Port one track first (mlc-compatible) into this framework by wiring:

1. model plugin: phiva
2. dataset plugin: vqav2
3. deployment plugin: mlc_compatible
4. runtime backend plan: mlc first, then llama_cpp for comparison