Skip to main content

multimodal-mcp

One MCP server for image generation and editing, speech recognition, text recognition in documents and images, speech synthesis and voice design, video generation, and multimodal agents that analyze images, audio and video. Declare the models and their capabilities in a config file; the server registers a tool only when a configured model supports it, and picks the model for each call from what the call needs.

Installation

The server requires Python 3.12 or later and runs over stdio:

uvx multimodal-mcp

or install it with pip install multimodal-mcp and run multimodal-mcp. For the development version, run uvx --from git+https://github.com/Vigilans/multimodal-mcp multimodal-mcp.

Register it with your MCP client as a stdio server named multimodal, for example:

{
  "mcpServers": {
    "multimodal": {
      "command": "uvx",
      "args": ["multimodal-mcp", "--config", "/home/me/.agents/mcp/multimodal/config.toml"]
    }
  }
}

The skill in skills/multimodal tells agents how to choose models, write prompts, check results and deliver them. Install it with npx skills add Vigilans/multimodal-mcp -s multimodal, adding -g to install it globally.

On Windows, omni agents run their commands in Git Bash, so they require Git for Windows with git on PATH.

Configuration

The config file is read from --config <path>, then MULTIMODAL_MCP_CONFIG, then ~/.agents/mcp/multimodal/config.toml.

[providers.gateway]
endpoint = "openai"
base_url = "https://gateway.example.com/v1"
env_key = "GATEWAY_API_KEY"   # or: api_key = "..."

[providers.gemini]
endpoint = "gemini"           # base_url omitted: the official API
env_key = "GEMINI_API_KEY"

[[models.image]]              # models.<image|asr|ocr|tts|video|omni>
provider = "gateway"
model = "gpt-image-2.5-flare"
description = "Fast image generation and editing"
capabilities = ["generate", "edit", "reference", "mask"]

[[models.video]]
provider = "gateway"
model = "MiniMax-H3"
description = "Video with synchronized audio; text, first/last frame, or reference driven"
capabilities = ["text", "frames"]

[[models.tts]]                # capabilities from the API notes
provider = "gemini"
model = "gemini-3.8-flash-tts"

[[models.omni]]               # omni models use an openai endpoint
provider = "gateway"
model = "gemini-3.5-flash"
capabilities = ["image", "audio"]

[defaults]                    # optional, per category
image = "gpt-image-2.5-flare"

A model without capabilities takes the list from api-notes.toml for its endpoint and model ID, and the config fails to load when the API notes have none; the server treats unlisted capabilities as unsupported. Capabilities in the config must match what the endpoint implements for that model, since the server sends every call the listed capabilities allow. Tools, list_models and defaults name a model by its model value, or by <provider>/<model> when several entries share that value, such as gateway/gpt-image-2.5-flare beside openai/gpt-image-2.5-flare.

Capabilities

Category Capability Meaning
image generate, edit Create an image from a prompt; modify given images
reference Generate a new image following a reference's subject or style
mask Edit only a masked region
asr text Plain transcription
segment, word Timestamp granularity
speaker Speaker labels
hotwords A dedicated hotword field: vLLM hotwords, Gemini custom_vocabulary
ocr text The text of each page
layout Labeled text blocks with their positions
tts voice Speak with a preset or registered voice ID
reference Imitate a voice sample sent with the request
timestamps Return segment and word timing of the generated speech
design Create a reusable voice ID from a description
video text Text-only generation
frames First frame, or first and last frames
reference Reference images, videos or audio
omni image, audio The media the model reads

Tools and results

Tool Registered when
list_models(category?, capability?) always
get_model_guide(model, lang, update?, api_notes?, prompt_guide?) always
generate_image an image model has generate
edit_image an image model has edit
transcribe_audio an asr model has text
transcribe_audio_with_timestamps an asr model has segment or word
recognize_text an ocr model has text
recognize_text_with_layout an ocr model has layout
generate_speech a tts model has voice or reference
voice_design a tts model has design
generate_video, get_video_job a video model exists; both list every video model
start_omni_agent, continue_omni_agent, get_omni_agent, wait_omni_agents, stop_omni_agent an omni model exists

get_model_guide returns a model's API notes from api-notes.toml and its family's prompt guide from prompt-guides.toml. The vendor reference files a guide names are downloaded into ~/.cache/multimodal-mcp/references/<family>/ on first use. Existing files are reused for either lang; update=true downloads them again in the requested lang, and a failed download keeps the existing file and returns an error. A deleted file is downloaded again on the next call.

Each tool's model parameter is an enum of the models that can perform it. Call arguments add requirements: references needs reference, mask needs mask, granularity: "word" needs word, speakers needs speaker, hotwords needs hotwords, voice and reference_audio need voice and reference, timestamps needs timestamps; a video with neither frames nor references needs text, first_frame and last_frame need frames, reference_images, reference_videos and reference_audios need reference. A named model that lacks a requirement is rejected before any request. Without model, the server uses the only capable model, else the category default if capable, else asks for a choice. Results report the model used.

generate_image, edit_image, transcribe_audio, transcribe_audio_with_timestamps, recognize_text, recognize_text_with_layout, generate_speech and generate_video take options, merged into the endpoint's request body, for fields beyond the common parameters; the API notes list the useful ones.

Results are written to ~/.cache/multimodal-mcp/<YYYY-MM-DD>/<category>/ unless an absolute output_path is given, and tools return the file path. generate_video submits a job; get_video_job saves the video when the job completes, to its own output_path when given. Timestamped transcription and generate_speech(timestamps=true) write {text, duration?, segments: [{start, end, text, speaker?}], words: [{start, end, word, speaker?}]} with times in seconds, where either list may be empty. recognize_text writes the pages to Markdown, each opened by <!-- page N -->; recognize_text_with_layout writes {page_count, pages: [{page, complete, blocks: [{label, box, text}]}]}, where box is [x1, y1, x2, y2] scaled to 0-1000 of the page's width and height. Both save the provider's response beside the result and return the pages whose text was cut short.

Omni agents work in the background: start_omni_agent returns an agent_id at once, wait_omni_agents waits until an agent's run ends, and get_omni_agent returns its answer and a page of the tool calls it made. continue_omni_agent adds a message to the active run, or starts a new run after the previous one ended; stop_omni_agent cancels the current run. Agents live in server memory and are lost when the server restarts, and a provider request already in flight may complete and be billed after cancellation. They read files and run shell commands such as ffmpeg and ffprobe, taken from PATH or, when PATH lacks either, downloaded once with static-ffmpeg.

Endpoints

base_url is the API root to which the endpoint appends its paths. When omitted, the endpoint's official API is used.

Endpoint Default base_url Image ASR OCR TTS Video
openai https://api.openai.com/v1 /images/generations, /images/edits /audio/transcriptions /audio/speech /videos, multipart, as served by vLLM-Omni
gemini https://generativelanguage.googleapis.com/v1beta /interactions /interactions /interactions, /voices /interactions, background
minimax https://api.minimaxi.com /v1/image_generation (references are portrait subjects) /v1/speech_to_text /v1/t2a_v2, /v1/voice_design /v2/video_generation
ark https://ark.cn-beijing.volces.com/api/v3 for image and video, https://openspeech.bytedance.com/api/v3 for speech /images/generations /sauc/bigmodel_nostream WebSocket /tts/unidirectional /contents/generations/tasks
deepseek https://api.deepseek.com /v1/tools/asr /v1/tools/ocr /v1/tools/tts

Omni models use the openai endpoint's chat completions. ark and deepseek ASR decode audio with ffmpeg from PATH, downloaded once with static-ffmpeg when PATH lacks it, the same copy omni agents use.

An ark base_url replaces both default roots (media and speech), so a provider with separate image/video and speech roots needs one provider entry per root. Include any plan prefix, such as .../api/plan/v3 for Agent Plan images and video and .../api/v3/plan for Agent Plan speech. Ark speech models select their service with X-Api-Resource-Id: doubao-seed-tts-2.0 and doubao-seed-asr-2.0 map to seed-tts-2.0 and volc.seedasr.sauc.duration; any other model name is sent as the resource ID.

DeepSeek's speech and OCR requests name no model; configure them as deepseek-asr, deepseek-tts and deepseek-ocr, which take their capabilities from the API notes.

minimax and ark video inputs pass URLs, data: URIs and asset URIs such as asset://ID through unchanged and inline local files as base64. Ark accepts reference videos only as public URLs or asset:// IDs.

Development

uv sync --locked
uvx ruff check src/multimodal_mcp tests
uv run pytest
uv build

CI runs the tests on Linux and Windows. Changes to how omni agents run shell commands should also pass uv run pytest tests/test_workspace.py on Windows.

License

MIT

Metadata

Release files for multimodal-mcp 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for multimodal-mcp 0.3.0
File Size Uploaded Provenance Actions
multimodal_mcp-0.3.0.tar.gz 63.0 kB With provenance Details for multimodal_mcp-0.3.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for multimodal-mcp 0.3.0
File Interpreter ABI Platform Actions
multimodal_mcp-0.3.0-py3-none-any.whl Python 3 none any Details for multimodal_mcp-0.3.0-py3-none-any.whl

Total release size: 135.3 kB

Release files / multimodal_mcp-0.3.0.tar.gz

Download URL multimodal_mcp-0.3.0.tar.gz
Size 63.0 kB
Tags Source
SHA-256 checksum
How to use checksums
0c0e512522eca4ce94a878d9e624931363ad723cb3218db495c300ac2ad512de
BLAKE2b-256 checksum
How to use checksums
dd404e8ef4af64c12e559e136c0150c8275e6b26ad663aa403fb5f896c141ee1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Release files / multimodal_mcp-0.3.0-py3-none-any.whl

Download URL multimodal_mcp-0.3.0-py3-none-any.whl
Size 72.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
816763583c12cdf52896f713311e9bcc6bd39f9b44af2dda76d2d7fd010b5ec3
BLAKE2b-256 checksum
How to use checksums
281d2be8a5d8ef590b178a9e0fc5a05e725a6668897b1f23cf601b89b2d63aab
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

Security for multimodal-mcp 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for multimodal_mcp-0.3.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 14, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page