multimodal-maestro

👋 hello

Multimodal-Maestro gives you more control over large multimodal models to get the outputs you want. With more effective prompting tactics, you can get multimodal models to do tasks you didn't know (or think!) were possible. Curious how it works? Try our HF space!

🚧 The project is still under construction and the API is prone to change.

💻 install

Pip install the supervision package in a 3.11>=Python>=3.8 environment.

pip install multimodal-maestro

🚀 examples

GPT-4 Vision

Find dog.

>>> The dog is prominently featured in the center of the image with the label [9].

👉 read more

load image
```
import cv2

image = cv2.imread("...")
```

create and refine marks

import multimodalmaestro as mm

generator = mm.SegmentAnythingMarkGenerator(device='cuda')
marks = generator.generate(image=image)
marks = mm.refine_marks(marks=marks)

visualize marks

mark_visualizer = mm.MarkVisualizer()
marked_image = mark_visualizer.visualize(image=image, marks=marks)

image-vs-marked-image

prompt

prompt = "Find dog."

response = mm.prompt_image(api_key=api_key, image=marked_image, prompt=prompt)

>>> "The dog is prominently featured in the center of the image with the label [9]."

extract related marks

masks = mm.extract_relevant_masks(text=response, detections=refined_marks)

>>> {'6': array([
...     [False, False, False, ..., False, False, False],
...     [False, False, False, ..., False, False, False],
...     [False, False, False, ..., False, False, False],
...     ...,
...     [ True,  True,  True, ..., False, False, False],
...     [ True,  True,  True, ..., False, False, False],
...     [ True,  True,  True, ..., False, False, False]])
... }

multimodal-maestro

🚧 roadmap

Documentation page.
Segment Anything guided marks generation.
Non-Max Suppression marks refinement.
LLaVA demo.

💜 acknowledgement

Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V by Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao.

🦸 contribution

We would love your help in making this repository even better! If you noticed any bug, or if you have any suggestions for improvement, feel free to open an issue or submit a pull request.

Metadata

Release files for multimodal-maestro 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for multimodal-maestro 0.1.0
File	Size	Uploaded
multimodal_maestro-0.1.0.tar.gz	10.7 kB	Nov 29, 2023	Details

Built distribution (wheel)

Table of built distributions (wheels) for multimodal-maestro 0.1.0
File	Interpreter	ABI	Platform	Reset
multimodal_maestro-0.1.0-py3-none-any.whl 11.9 kB Nov 29, 2023	Python 3	none	any	Details

Total release size: 22.7 kB

Release files / multimodal_maestro-0.1.0.tar.gz

Download URL	multimodal_maestro-0.1.0.tar.gz
Size	10.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`799bea920f212f8ba1aeb42ca7473efae1ebff4c6185b1d7593c0887c2a61252`
BLAKE2b-256 checksum How to use checksums	`06096ef2e07b42b9659cd62e82b07e22e8ed3a26c5884cd843b581715a9cdb78`
Upload date	Nov 29, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.5.1 CPython/3.10.13 Darwin/23.0.0`

Release files / multimodal_maestro-0.1.0-py3-none-any.whl

Download URL	multimodal_maestro-0.1.0-py3-none-any.whl
Size	11.9 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`10836ab4608a4e32108b8cc174352b6ab66b816711dd55fd02a56ad68ecbf40b`
BLAKE2b-256 checksum How to use checksums	`c0e319463106d1c459fb981418dc71b7f2c19ecbd799915bb1047db14425457e`
Upload date	Nov 29, 2023
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.5.1 CPython/3.10.13 Darwin/23.0.0`