imageforllm 0.4.0

A library for embedding code and plot properties into matplotlib images for LLM context

ImageForLLM 🖼️

English | 中文

A free lunch for LLM recognition images 🍱

Overview 🔍

ImageForLLM enables embedding source comment and plot properties into matplotlib images, particularly useful when sharing plots with Large Language Models (LLMs). This allows the LLM to understand how the plot was generated and what it represents. It also supports adding metadata to AI-generated images for better context.

Easyuse ✨

With just TWO lines, it can automatically add the information of the generated image in the metadata of the image generated by matplotlib for you without any additional operations.

For LLM or other readers, the content of the picture can be quickly understood through this information.

import imageforllm
imageforllm.hook_image_save()

Installation 📦

pip install imageforllm

The package requires Pillow for metadata embedding:

pip install Pillow

Features ✅

• Embed source comment that generated a plot into the image metadata
• Automatically extract and embed plot properties (titles, labels, etc.)
• Add AI generation metadata (model, prompt, parameters) to AI-generated images
• Extract embedded comment, properties, and AI metadata from images
• Command-line tools for extracting metadata from images and adding AI metadata
• Get all metadata as JSON for easy integration with other tools
• NEW: Direct support for working with image byte objects (from base64 or URLs) via dedicated functions

Usage 🚀

Basic Workflow for Matplotlib

import matplotlib.pyplot as plt
import numpy as np
import imageforllm

# 1. Hook matplotlib's savefig function
imageforllm.hook_image_save()

# 2. Define your plot comment as a string
# This is not necessary.
plot_source_comment = """
It make work for a wave plot.
"""

# 3. Create your plot
x = np.linspace(0, 10, 100)
y = np.sin(x)
plt.plot(x, y)
plt.title('Sine Wave')
plt.xlabel('Time')
plt.ylabel('Amplitude')

# 4. Save with embedded comment and auto-extracted properties
plt.savefig('sine_wave_plot.png', create_comment=plot_source_comment)

# 5. (Optional) Unhook when done
imageforllm.unhook_image_save()

Adding AI Metadata to Images

import imageforllm

# Add AI metadata to an existing image file
model = "stable-diffusion-xl-1.0"
prompt = "A serene mountain landscape with a lake reflecting the sunset"
parameters = {
    "seed": 42,
    "guidance_scale": 7.5,
    "num_inference_steps": 50
}

imageforllm.add_ai_metadata('ai_generated_image.png', model, prompt, parameters)

Working with Image Bytes Objects

For direct work with image bytes (from base64, URLs, etc.), you can use the dedicated bytes-specific functions:

import imageforllm
import base64
import requests
import io

# Example 1: Working with base64 encoded images
base64_data = "..." # your base64 encoded image data
img_bytes = base64.b64decode(base64_data)

# Extract metadata from bytes using bytes-specific function
metadata = imageforllm.get_all_metadata_json_from_bytes(img_bytes)
print(metadata)

# Add metadata to bytes and get BytesIO result
result_buffer = imageforllm.add_ai_metadata_to_bytes(
    img_bytes,
    model="stable-diffusion-xl",
    prompt="A city skyline",
    parameters={"seed": 123}
)
# result_buffer is a BytesIO object you can use directly
modified_bytes = result_buffer.getvalue()

# Save the modified bytes to a file if needed
with open("output_image.png", "wb") as f:
    f.write(modified_bytes)

# Example 2: Working with images from URLs
response = requests.get("https://github.com/kexinoh/imageforllm/blob/main/examples/aigirl_with_metadata.png?raw=true")
img_data = response.content
ai_metadata = imageforllm.extract_ai_metadata_from_bytes(img_data)
print(ai_metadata)

Extracting Metadata 🔄

import imageforllm

# Get all metadata from an image as a JSON-serializable dictionary
all_info = imageforllm.get_all_metadata_json('image.png')

# Access embedded comment
comment = all_info.get('source_comment')
print(comment)

# Access plot properties
properties = all_info.get('plot_properties')
print(properties)

# Access AI metadata
ai_model = all_info.get('ai_model')
prompt = all_info.get('prompt')
parameters = all_info.get('parameters')
print(f"Model: {ai_model}, Prompt: {prompt}")

# Extract only AI-specific metadata
ai_metadata = imageforllm.extract_ai_metadata('ai_generated_image.png')
print(ai_metadata)

Command-line Extraction 🖥️

The package includes command-line tools for working with metadata:

# Extract and print comment
python -m imageforllm.extract image.png

# Extract and print all metadata
python -m imageforllm.extract image.png --info

# Extract only plot properties
python -m imageforllm.extract image.png --properties

# Extract only AI metadata
python -m imageforllm.extract image.png --ai

# Output in JSON format
python -m imageforllm.extract image.png --json

# Save extracted comment to a file
python -m imageforllm.extract image.png -o extracted_comment.py

Adding AI Metadata via Command Line

# Add AI metadata to an image
python -m imageforllm.ai_metadata image.png --model "stable-diffusion" --prompt "mountain landscape" --parameters '{"seed": 42}'

Limitations ⚠️

• Metadata embedding is primarily supported for PNG format
• When saving to file-like objects, metadata embedding is not supported
• The package cannot automatically determine the comment that generated a plot; you must provide it as a string

How It Works 🔧

1. The package hooks matplotlib's savefig function
2. When saving, it captures any provided source comment and automatically extracts plot properties
3. It embeds this metadata into the PNG image using Pillow
4. For AI-generated images, you can add model, prompt, and parameter information
5. Metadata can later be extracted from the image using the provided functions or command-line tools

Example 📝

See the included examples:

• examples/saveandread.py - saving and reading metadata
• examples/ai_metadata_example.py - working with AI-generated images
• examples/bytes_example.py - working with image bytes directly

API Reference 📚

Main Functions

File-Based Functions

• hook_image_save(): Replaces matplotlib's savefig with a version that embeds metadata
• unhook_image_save(): Restores the original savefig function
• get_image_info(image_path): Extracts metadata from an image file
• get_all_metadata_json(image_path): Gets all ImageForLLM-specific metadata as a JSON-serializable dictionary
• add_ai_metadata(image_path, model, prompt, parameters=None): Adds AI generation metadata to an image file
• extract_ai_metadata(image_path): Extracts only AI-specific metadata from an image file

Bytes-Based Functions

• get_image_info_from_bytes(image_bytes): Extracts metadata from image bytes
• get_all_metadata_json_from_bytes(image_bytes): Gets all metadata from image bytes as a JSON-serializable dictionary
• add_ai_metadata_to_bytes(image_bytes, model, prompt, parameters=None): Adds AI generation metadata to image bytes and returns a BytesIO object
• extract_ai_metadata_from_bytes(image_bytes): Extracts only AI-specific metadata from image bytes

Constants

• METADATA_KEY_COMMENT: Key for source comment in metadata dictionary
• METADATA_KEY_PROPERTIES: Key for plot properties in metadata dictionary
• METADATA_KEY_AI_MODEL: Key for AI model information
• METADATA_KEY_PROMPT: Key for generation prompt
• METADATA_KEY_PARAMETERS: Key for additional parameters

License 📄

MIT