multimodal-agent 0.7.0

A Python multimodal agent for interacting with Gemini models via text, images, and CLI.

Multimodal-Agent

A lightweight, production-ready multimodal wrapper for Google Gemini with RAG, image input, JSON mode, project learning, session memory, and a clean CLI & server.

---

Features

Core LLM Capabilities

• Unified agent for text, image, and chat interactions
• Clean CLI : agent ask, agent image, agent chat, agent history, agent learn-project
• Supports Gemini 2.5-flash , 1.5-flash , and any future model (configurable)
• Automatic retry logic with exponential backoff
• Full offline mode support (FAKE_RESPONSE) when no API key is available
• Detailed usage logging : prompt, response, and total token counts

RAG + Memory

• Local SQLite RAGStore (no cloud dependency)
• Automatic memory saving of past chats
• Project learning: let the agent read source code & architecture
• Project introspection commands: learn-project, show-project, inspect-project

Configuration System

• User config stored at: ~/.multimodal_agent/config.yaml
• Configure models individually:
  • chat_model
  • image_model
  • embedding_model
• New CLI commands:
  • agent config set-model <model>
  • agent config set-image-model <model>
  • agent config set-embed-model <model>
  • agent config set-key <API_KEY>

Developer Experience

• pytest fixtures for offline/fake mode
• High test coverage rate
• Type-safe AgentResponse
• Extensible architecture
• Easy to embed into apps or scripts

---

Installation

pip install multimodal-agent

Or local:

pip install -e .

---

Configuration

Show current configuration:

agent config show

Set API key:

agent config set-key YOUR_KEY

Set chat model:

agent config set-model gemini-2.5-flash

Set image model:

agent config set-image-model gemini-1.5-flash

Set embedding model:

agent config set-embed-model text-embedding-004

Your config file after updates:

local_learning: true
chat_model: gemini-2.5-flash
image_model: gemini-2.0-flash
embedding_model: text-embedding-004
api_key: YOUR_KEY

---

Quick Start

Text Question

agent ask "What is the capital of France?"

Disable RAG

agent ask "What is the capital of France?" --no-rag

JSON mode

agent ask "give me json" --json

Image + Text

agent image test.jpg "describe this"

Chat (with persistent memory)

agent chat

---

History / Memory

Your memory DB lives at:

~/.multimodal_agent/memory.db

Show memory:

agent history show

Clear memory:

agent history clear

Summarize memory:

agent history summary

---

Learning a Project

Let the agent scan and store a project summary:

agent learn-project my_app/

List learned projects:

agent list-projects

Show a specific project:

agent show-project project:my_app

Inspect project without saving:

agent inspect-project my_app/

---

Python API Example

from multimodal_agent.core.agent_core import MultiModalAgent

agent = MultiModalAgent()

resp = agent.ask("Explain quantum computing")
print(resp.text)
print(resp.usage)

Image example:

from multimodal_agent.utils import load_image_as_part

img = load_image_as_part("cat.jpg")
resp = agent.ask_with_image("describe this", img)
print(resp.text)

---

Server Mode

Start:

agent server

Runs at:

http://127.0.0.1:8000

API Reference (v0.6.0)

POST /ask

curl -X POST http://127.0.0.1:8000/ask \
  -H "Content-Type: application/json" \
  -d '{"prompt": "hello"}'

Response:

{
  "text": "hello",
  "data": null,
  "usage": { "prompt_tokens": 44, "response_tokens": 3, "total_tokens": 553 }
}

POST /ask_with_image

curl -X POST http://127.0.0.1:8000/ask_with_image \
  -F "file=@test.jpg" \
  -F "prompt=describe this"

v0.6.0 Better Error Handling

Failures now return:

{
  "text": "Image processing failed: 429 RESOURCE_EXHAUSTED ...",
  "data": null,
  "usage": {},
  "error": true
}

Never returns text: null.

---

POST /generate

curl -X POST http://127.0.0.1:8000/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt": "give me json", "json": true}'

---

POST /memory/search

curl -X POST http://127.0.0.1:8000/memory/search \
  -H "Content-Type: application/json" \
  -d '{"query": "hello"}'

Response:

{
  "results": [
    [0.98, { "id": 1, "content": "hello", "role": "user" }]
  ]
}

---

POST /learn/project

Returns a structured project profile:

{
  "status": "ok",
  "project_id": "project:rope_simulation_using_flutter",
  "profile": {
    "package_name": "rope_simulation_using_flutter",
    "architecture": {
      "patterns": ["feature_first"],
      "state_management": []
    },
    "dart_files_count": 3,
    "widget_files_count": 2
  }
}

---

Architecture Overview

multimodal_agent/
    core/          # Main agent logic
    rag/           # SQLite vector store
    cli/           # CLI commands (`agent`)
    server/        # FastAPI server implementation
    utils/         # helpers

Memory schema:

sessions      # chat sessions
chunks        # tokenized fragments
embeddings    # vector embeddings
projects      # project profiles (v0.6.0)

---

Formatting Engine (v0.4.0+)

• Detects JSON, XML, HTML, code, python, kotlin, dart, js, swift …
• Pretty-prints output
• Auto-wraps in fenced code blocks
• Optional in agent.ask(formatted=True)

---

Running Tests

make test
make coverage

This includes:

• RAG tests
• CLI tests
• JSON mode tests
• Fake mode (offline)
• Config isolation
• SQLite operations

---

Roadmap

v0.8.0

• Streaming responses
• Conversations with images
• Project-diff memory updates

v1.0

• Stable API
• Plugin ecosystem
• Multi-language project analyzers

---

License

MIT License.