StableI2I

Official implementation of StableI2I: Spotting Unintended Changes in Image-to-Image Transition (ICML 2026)

Any questions can be consulted -> (Email:lijiayang.cs@gmail.com)

Looking forward to your ⭐！

📌 TODOs

• release code
• release ckpt
• release pip-pkg
• release arxiv
• ICML version paper

Core Concept:

In most real-world image-to-image (I2I) scenarios, existing evaluations primarily focus on instruction following and the perceptual quality or aesthetics of the generated images. However, they largely fail to assess whether the output image preserves the semantic correspondence and spatial structure of the input image. To address this limitation, we propose StableI2I, a unified and dynamic evaluation framework that explicitly measures content fidelity and pre--post consistency across a wide range of I2I tasks without requiring reference images, including image editing and image restoration. In addition, we construct StableI2I-Bench, a benchmark designed to systematically evaluate the accuracy of MLLMs on such fidelity and consistency assessment tasks. Extensive experimental results demonstrate that StableI2I provides accurate, fine-grained, and interpretable evaluations of content fidelity and consistency, with strong correlations to human subjective judgments. Our framework serves as a practical and reliable evaluation tool for diagnosing content consistency and benchmarking model performance in real-world I2I systems.

Python Package

Install from PyPI:

pip install stablei2i

From source:

pip install -e .
# or
pip install git+https://github.com/Henry-Lee-real/StableI2I.git

The evaluation prompt templates are bundled in the package. Callers only pass the input image, output image, and task prompt.

Load the judge once, then reuse it for single-pair eval, JSONL eval, or online RL rewards:

from stablei2i import StableI2I

judge = StableI2I(model="lijiayangCS/StableI2I_PLUS")  # or a local ckpt path

# single pair (path / PIL / numpy / torch tensor)
result = judge.evaluate(
    input_image="before.png",
    output_image="after.png",
    prompt="Add a wooden bench along the path.",
    mode="cot",  # simple | cot | score
)

# jsonl batch
rows = judge.evaluate_jsonl("test_jsonl/sample.jsonl", output_jsonl="outputs/results.jsonl")

# online RL reward: 0-10 by default, or normalize to [0, 1]
reward = judge.reward(before, after, prompt, normalize=True)
reward_fn = judge.as_reward_fn(normalize=True)  # callable(input, output, prompt) -> float

CLI (same interface as test.py):

stablei2i --input-image before.png --output-image after.png --prompt "Restore the image." --mode score --ckpt lijiayangCS/StableI2I_PLUS
stablei2i --jsonl test_jsonl/sample.jsonl --ckpt path/to/ckpt --mode cot --output-jsonl outputs/results.jsonl

See infer.md for JSONL format, modes, and RL notes.

Environment Setting:

Install dependencies:

pip install -r requirements.txt

The specific environment is consistent with that of Qwen3-VL. For the library API, pip install stablei2i is enough.

APP Usage:

app.py is the local web demo and API entry. Running it starts a FastAPI service with a browser UI.

Example:

set MODEL_PATH=path/to/ckpt
set GPU_ID=0
set HOST=127.0.0.1
set PORT=10004
python app.py

Then open:

http://127.0.0.1:10004

The demo supports:

• built-in examples
• inference by local image path
• inference by image upload
• summarized semantic / structure / low-level results

Inference

See infer.md.

Training

Recommended official references:

• Qwen3-VL: QwenLM/Qwen3-VL
• Swift: modelscope/ms-swift

Notes:

• For SFT, start from the official Qwen3-VL finetuning workflow.
• For GRPO and related alignment training, use Swift.

Citation

If you find our work helpful for your research, please consider citing our work.

@article{li2026stablei2i,
  title={StableI2I: Spotting Unintended Changes in Image-to-Image Transition},
  author={Li, Jiayang and Cao, Shuo and Li, Xiaohui and Zhang, Zhizhen and Zhu, Kaiwen and Duan, Yule and Qiao, Yu and Zhang, Jian and Liu, Yihao},
  journal={arXiv preprint arXiv:2605.04453},
  year={2026}
}