👋 hello
Multimodal-Maestro gives you more control over large multimodal models to get the outputs you want. With more effective prompting tactics, you can get multimodal models to do tasks you didn't know (or think!) were possible. Curious how it works? Try our HF space!
🚧 The project is still under construction and the API is prone to change.
💻 install
Pip install the supervision package in a 3.11>=Python>=3.8 environment.
pip install multimodal-maestro
🚀 examples
GPT-4 Vision
Find dog.
>>> The dog is prominently featured in the center of the image with the label [9].
👉 read more
-
load image
import cv2 image = cv2.imread("...")
-
create and refine marks
import multimodalmaestro as mm generator = mm.SegmentAnythingMarkGenerator(device='cuda') marks = generator.generate(image=image) marks = mm.refine_marks(marks=marks)
-
visualize marks
mark_visualizer = mm.MarkVisualizer() marked_image = mark_visualizer.visualize(image=image, marks=marks)
-
prompt
prompt = "Find dog." response = mm.prompt_image(api_key=api_key, image=marked_image, prompt=prompt)
>>> "The dog is prominently featured in the center of the image with the label [9]." -
extract related marks
masks = mm.extract_relevant_masks(text=response, detections=refined_marks)
>>> {'6': array([ ... [False, False, False, ..., False, False, False], ... [False, False, False, ..., False, False, False], ... [False, False, False, ..., False, False, False], ... ..., ... [ True, True, True, ..., False, False, False], ... [ True, True, True, ..., False, False, False], ... [ True, True, True, ..., False, False, False]]) ... }
🚧 roadmap
- Documentation page.
- Segment Anything guided marks generation.
- Non-Max Suppression marks refinement.
- LLaVA demo.
💜 acknowledgement
- Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V by Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao.
🦸 contribution
We would love your help in making this repository even better! If you noticed any bug, or if you have any suggestions for improvement, feel free to open an issue or submit a pull request.
Metadata
Release files for multimodal-maestro 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| multimodal_maestro-0.1.0.tar.gz | 10.7 kB | Without provenance | Details for multimodal_maestro-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| multimodal_maestro-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for multimodal_maestro-0.1.0-py3-none-any.whl |
Total release size: 22.7 kB
Release files / multimodal_maestro-0.1.0.tar.gz
| Download URL | multimodal_maestro-0.1.0.tar.gz |
|---|---|
| Size | 10.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
799bea920f212f8ba1aeb42ca7473efae1ebff4c6185b1d7593c0887c2a61252
|
|
BLAKE2b-256 checksum How to use checksums |
06096ef2e07b42b9659cd62e82b07e22e8ed3a26c5884cd843b581715a9cdb78
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.5.1 CPython/3.10.13 Darwin/23.0.0
|
Release files / multimodal_maestro-0.1.0-py3-none-any.whl
| Download URL | multimodal_maestro-0.1.0-py3-none-any.whl |
|---|---|
| Size | 11.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
10836ab4608a4e32108b8cc174352b6ab66b816711dd55fd02a56ad68ecbf40b
|
|
BLAKE2b-256 checksum How to use checksums |
c0e319463106d1c459fb981418dc71b7f2c19ecbd799915bb1047db14425457e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.5.1 CPython/3.10.13 Darwin/23.0.0
|
Security for multimodal-maestro 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.