abstractvision 0.3.16

Model-agnostic generative vision abstractions (image/video) for the Abstract ecosystem

AbstractVision

Model-agnostic generative vision API (images, optional video) for Python and the Abstract* ecosystem.

What you get

• A small orchestration API: VisionManager
• A packaged capability registry (“what models can do”): VisionModelCapabilitiesRegistry backed by vision_model_capabilities.json
• Shared model metadata that now also drives local catalog surfacing and backend request normalization across the CLI, playground, and AbstractCore paths
• Optional artifact-ref outputs (small JSON refs): LocalAssetStore and RuntimeArtifactStoreAdapter
• Built-in backends (execution engines): src/abstractvision/backends/
  • OpenAI-compatible HTTP: openai_compatible.py
  • Local Diffusers: huggingface_diffusers.py
  • Local stable-diffusion.cpp / GGUF: stable_diffusion_cpp.py
  • Local MLX-Gen Apple Silicon backend for curated AbstractFramework MLX presets: mflux.py
• CLI for manual testing (abstractvision cli, legacy alias: abstractvision repl): abstractvision
• Self-contained local Playground UI/API: playground/vision_playground.html (docs: playground/README.md)

How it fits together (diagram)

flowchart LR
  Caller[Python / CLI / AbstractCore] --> VM[VisionManager]
  VM --> BE[VisionBackend]
  BE --> VM
  VM -->|optional| Store[MediaStore]
  Store --> Ref[Artifact ref dict]
  VM -->|no store| Asset["GeneratedAsset (bytes + mime)"]

Status (current backend support)

• Development status: Alpha (0.x). The public API is stable-by-design, but breaking changes may still happen and will be called out in CHANGELOG.md.
• Built-in backends implement images (text_to_image, image_to_image) plus backend-dependent video (text_to_video, image_to_video).
• Local MLX-Gen supports text_to_image for curated FLUX.2, Qwen Image, Z-Image, ERNIE Image Turbo, and FIBO models; supports image_to_image for FLUX.2 klein/base, Qwen Image Edit, ERNIE Image Turbo, FIBO, and FIBO Edit models; and supports Wan 2.2 TI2V for local text_to_video and first-frame image_to_video.
• Local Diffusers text_to_video remains experimental and is temporarily disabled from the normal local runtime surfaces pending docs/backlog/planned/0023_local_runtime_capability_quarantine_for_glm_mflux_and_t2v.md.
• Remote text_to_video / image_to_video are also supported through the OpenAI-compatible backend when endpoints are configured.
• multi_view_image is part of the public API (VisionManager.generate_angles) but no built-in backend implements it yet.

Details: docs/reference/backends.md.

Installation

pip install abstractvision

The base install is lightweight. It includes the shared API, capability registry, artifact helpers, CLI, AbstractCore plugin entry point, and the stdlib OpenAI-compatible HTTP backend. Local inference runtimes are explicit extras.

Optional extras:

Extra	Use
abstractvision[openai]	Official OpenAI provider intent marker; no SDK dependency today.
abstractvision[openai-compatible]	Generic local/remote OpenAI-shaped endpoint intent marker; stdlib-only today.
abstractvision[models]	Curated Hugging Face download helpers for cache-backed local quantized vision model presets.
abstractvision[diffusers]	Install Torch/Diffusers and related packages for local Diffusers generation.
abstractvision[huggingface]	Compatibility alias for callers that still request the historical Diffusers extra.
abstractvision[sdcpp]	Install stable-diffusion-cpp-python for the pip binding fallback.
abstractvision[mlx-gen]	Install the optional MLX-Gen Apple Silicon image/video runtime.
abstractvision[mflux]	Compatibility alias for the MLX-Gen Apple Silicon image/video runtime.
abstractvision[local]	Convenience for both local backend dependency sets, including diffusers and sdcpp.
abstractvision[all]	All runtime backend dependencies, without contributor tooling.
abstractvision[apple] / abstractvision[all-apple]	Native macOS Python profile: Diffusers/Torch MPS, stable-diffusion.cpp bindings, and MLX-Gen.
abstractvision[gpu]	GPU Diffusers/Torch profile. Install a CUDA/ROCm-enabled PyTorch wheel when needed.
abstractvision[all-gpu]	Full GPU-relevant local vision profile: Diffusers plus stable-diffusion.cpp bindings.
abstractvision[abstractcore]	Compatibility marker only; AbstractCore is still supplied by the host application.

stable-diffusion-cpp-python is currently constrained below 0.4.6 because that release's source distribution is missing vendored CMake files required by native Linux builds.

Contributor-only extras:

Extra	Use
abstractvision[diffusers-dev] / abstractvision[huggingface-dev]	Looser dependency pins for newer/unreleased Diffusers pipelines; install Diffusers main separately if needed.
abstractvision[test]	Local test dependencies.
abstractvision[docs]	Documentation build tooling.
abstractvision[dev]	Full contributor workflow: tests, docs, build, lint, formatting, and pre-commit. Do not use this as an application runtime profile.

Note (CUDA): on Windows/Linux, pip install "abstractvision[diffusers]" may install a CPU-only PyTorch build. If you want to use an NVIDIA GPU, install a CUDA-enabled PyTorch build first (see https://pytorch.org/get-started/locally/) and verify torch.cuda.is_available() is True.

AbstractCore is not installed by AbstractVision. When an AbstractCore application has AbstractVision installed in the same environment, AbstractCore can discover the plugin entry point and use the integration modules lazily.

If you hit “missing pipeline class” errors for newer model families, see docs/getting-started.md. In that case you may need Diffusers from source (main):

pip install -U "abstractvision[diffusers-dev]"
pip install -U "git+https://github.com/huggingface/diffusers@main"

For local development from a repo checkout:

pip install -e ".[dev]"

Usage

Start here:

• Getting started: docs/getting-started.md
• FAQ: docs/faq.md
• Troubleshooting: docs/troubleshooting.md
• API reference: docs/api.md
• Architecture: docs/architecture.md
• Capability registry + catalog policy: docs/reference/capabilities-registry.md, docs/adr/README.md
• Docs index: docs/README.md

First Apple-local MLX model

For Apple Silicon local image generation, prefer the AbstractFramework MLX-Gen q4 presets first. They are published in the AbstractFramework/mlx-gen Hugging Face collection and are the default recommendation for local memory efficiency. q8 variants are also listed as separate model repos and should be selected explicitly when quality is more important than memory footprint. Qwen and ERNIE q4 prepared folders can mix q4 and q8 components, but they remain the default prepared choice.

The downloader stores curated presets in the Hugging Face cache by default and imports older ~/models/<preset> trees on first use. Generation stays cache-only unless you explicitly enable runtime downloads.

pip install "abstractvision[models,mlx-gen]"
abstractvision model-presets
abstractvision catalog --provider mlx-gen
# Tip: `--provider mlx-gen` implies `--target mlx` (you usually set one or the other).
abstractvision download AbstractFramework/flux.2-klein-4b-4bit --provider mlx-gen
abstractvision download AbstractFramework/qwen-image-2512-4bit --provider mlx-gen
abstractvision download AbstractFramework/qwen-image-edit-2511-4bit --provider mlx-gen
abstractvision download AbstractFramework/z-image-turbo-4bit --provider mlx-gen
abstractvision download AbstractFramework/ernie-image-turbo-8bit --provider mlx-gen
abstractvision download briaai/FIBO --provider mlx-gen
abstractvision download briaai/Fibo-lite --provider mlx-gen
abstractvision download briaai/Fibo-Edit --provider mlx-gen
abstractvision download Wan-AI/Wan2.2-TI2V-5B-Diffusers --provider mlx-gen
abstractvision t2i --provider mlx-gen --model AbstractFramework/flux.2-klein-4b-4bit "a product photo of a matte black espresso machine" --steps 4 --guidance-scale 1.0
abstractvision t2i --provider mlx-gen --model AbstractFramework/flux.2-klein-4b-8bit "a product photo of a matte black espresso machine" --steps 4 --guidance-scale 1.0
abstractvision i2i --provider mlx-gen --model AbstractFramework/qwen-image-edit-2511-4bit --image ./input.png "replace the background with a clean white studio setup" --steps 20 --guidance-scale 2.5 --strength 0.75
abstractvision t2i --provider mlx-gen --model briaai/FIBO "a studio product photo of a white ceramic mug with the AbstractFramework logo" --steps 50 --guidance-scale 4.0
abstractvision i2i --provider mlx-gen --model briaai/Fibo-Edit --image ./input.png "remove the background and keep the object edges clean" --steps 20 --guidance-scale 4.0
abstractvision t2v --provider mlx-gen --model Wan-AI/Wan2.2-TI2V-5B-Diffusers "a red fox walking through a snowy forest, cinematic" --frames 121 --fps 24 --steps 50 --guidance-scale 5.0
abstractvision i2v --provider mlx-gen --model Wan-AI/Wan2.2-TI2V-5B-Diffusers --image ./first-frame.png "slow camera push-in" --frames 121 --fps 24 --steps 50 --guidance-scale 5.0

Select q4 or q8 by using the exact published model id, for example AbstractFramework/flux.2-klein-4b-4bit or AbstractFramework/flux.2-klein-4b-8bit. Quantization is metadata of the published model folder, not a generation-time override.

The shipped MLX-Gen backend currently supports curated q4/q8 prepared folders for flux2-klein-4b, flux2-klein-9b, flux2-klein-base-4b, flux2-klein-base-9b, qwen-image, qwen-image-edit, z-image, and z-image-turbo families, plus the q4/q8 ernie-image-turbo prepared folders. MLX-Gen 0.18.6+ also runs official runtime snapshots such as briaai/FIBO, briaai/Fibo-lite, briaai/Fibo-Edit, briaai/Fibo-Edit-RMBG, and Wan-AI/Wan2.2-TI2V-5B-Diffusers. image_to_image is implemented for FLUX.2 klein/base, Qwen Image Edit, ERNIE Image Turbo, FIBO, and FIBO Edit models; FIBO Edit snapshots support mask inputs where the runtime supports them. Edit strength is passed as strength and normalized to MLX-Gen's image_strength parameter where the runtime supports it. Wan 2.2 TI2V uses the official snapshot through MLX-Gen 0.18.6+ for text_to_video and first-frame image_to_video; it is not an AbstractFramework q4/q8 prepared folder, so select it by its exact repo id.

One-shot t2i, i2i, t2v, and i2v commands store results in the local asset store and print an artifact ref followed by the local content path. Use --open when you want the generated output opened after storage, or --store-dir <dir> to choose the asset store directory. MLX-Gen video commands report frame/step progress on stderr by default; pass --no-progress when you need quiet output.

Stable Diffusion does not currently have a curated MLX-Gen q4/q8 preset in AbstractVision, so full Diffusers downloads remain explicit.

Install the Diffusers runtime extra, download a Diffusers snapshot, then select the Diffusers backend explicitly:

pip install "abstractvision[models,diffusers]"
abstractvision catalog --provider diffusers
# Tip: `--provider diffusers` implies `--target diffusers` (you usually set one or the other).
abstractvision download stable-diffusion --provider diffusers
abstractvision download sd1.4 --provider diffusers
abstractvision download sd1.5-inpaint --provider diffusers
abstractvision download sdxl-base --provider diffusers
abstractvision download sdxl-inpaint --provider diffusers
abstractvision download sd3-medium --provider diffusers
abstractvision download sd3.5-large --provider diffusers
abstractvision download ernie-image --provider diffusers
abstractvision download qwen-image-edit-2511 --provider diffusers
abstractvision download flux2-dev --provider diffusers
export ABSTRACTVISION_BACKEND=diffusers
export ABSTRACTVISION_MODEL_ID=runwayml/stable-diffusion-v1-5
export ABSTRACTVISION_DIFFUSERS_DEVICE=auto
abstractvision cli

Notes:

• abstractvision download qwen-image-edit-2511 --provider diffusers downloads the curated official 16-bit Diffusers snapshot.
• GLM-Image remains in the packaged registry, but local Diffusers GLM-Image is temporarily disabled pending the follow-up tracked in docs/backlog/planned/0023_local_runtime_capability_quarantine_for_glm_mflux_and_t2v.md.
• CogVideoX-2b downloads are still available for experimentation, but local text_to_video is currently marked experimental and disabled from the normal product surfaces.

For a fresh cache, you can also permit the interactive CLI to download missing files:

ABSTRACTVISION_DIFFUSERS_ALLOW_DOWNLOAD=1 abstractvision cli

More recommendations by VRAM: docs/getting-started.md.

Capability-driven model selection

from abstractvision import VisionModelCapabilitiesRegistry

reg = VisionModelCapabilitiesRegistry()
assert reg.supports("runwayml/stable-diffusion-v1-5", "text_to_image")
assert reg.supports("Qwen/Qwen-Image-Edit-2511", "image_to_image")

print(reg.list_tasks())
print(reg.models_for_task("text_to_image"))
print(reg.models_for_task("image_to_image"))

Backend wiring + generation (artifact outputs)

The base install is import-light and does not install Torch/Diffusers. Heavy local backend modules are imported lazily (see src/abstractvision/backends/__init__.py). Install abstractvision[diffusers] for local Diffusers, or abstractvision[sdcpp] for the optional stable-diffusion.cpp python binding fallback.

from abstractvision import LocalAssetStore, VisionManager, VisionModelCapabilitiesRegistry, is_artifact_ref
from abstractvision.backends import OpenAICompatibleBackendConfig, OpenAICompatibleVisionBackend

reg = VisionModelCapabilitiesRegistry()

backend = OpenAICompatibleVisionBackend(
    config=OpenAICompatibleBackendConfig(
        base_url="http://localhost:1234/v1",
        api_key="YOUR_KEY",      # optional for local servers
        model_id="REMOTE_MODEL", # optional (server-dependent)
    )
)

vm = VisionManager(
    backend=backend,
    store=LocalAssetStore(),         # enables artifact-ref outputs
    model_id="Qwen/Qwen-Image-Edit-2511",  # optional: capability gating
    registry=reg,                   # optional: reuse loaded registry
)

out = vm.generate_image("a cinematic photo of a red fox in snow")
assert is_artifact_ref(out)
print(out)  # {"$artifact": "...", "content_type": "...", ...}

png_bytes = vm.store.load_bytes(out["$artifact"])  # type: ignore[union-attr]

When installed next to AbstractCore, AbstractVision is also discovered as a llm.vision capability plugin. The plugin defaults to the official OpenAI image endpoint (https://api.openai.com/v1) and reads OPENAI_API_KEY. Set OPENAI_BASE_URL when you need a local or remote compatible /v1 server, and use the same OPENAI_API_KEY bearer token if that endpoint requires auth. Set ABSTRACTVISION_BACKEND=openai-compatible when you want to force compatible-endpoint semantics. Set ABSTRACTVISION_MODEL_ID, OPENAI_IMAGE_MODEL_ID, or OPENAI_IMAGE_MODEL when you need an explicit image model (static default OpenAI model: gpt-image-1). AbstractVision does not query provider /models catalogs to discover or select image models automatically, but you can inspect them explicitly with abstractvision provider-models, VisionManager.list_provider_models(...), or the AbstractCore plugin method llm.vision.list_provider_models(...). After inspection, set the model env var explicitly for newer provider models when available to your account. Set ABSTRACTVISION_BACKEND=mlx-gen, ABSTRACTVISION_BACKEND=diffusers, or ABSTRACTVISION_BACKEND=sdcpp when you want AbstractCore to launch local AbstractVision generation directly. For MLX-Gen, set ABSTRACTVISION_MFLUX_MODEL=AbstractFramework/flux.2-klein-4b-4bit or use routed model ids such as mlx-gen/AbstractFramework/flux.2-klein-4b-4bit. Legacy mflux provider values remain accepted as compatibility aliases.

Interactive testing (CLI)

abstractvision models
abstractvision provider-models --openai --task text_to_image
abstractvision provider-models --base-url http://localhost:1234/v1 --task text_to_image
abstractvision tasks
abstractvision show-model runwayml/stable-diffusion-v1-5

abstractvision cli

Inside the interactive CLI:

/t2i "a watercolor painting of a lighthouse" --width 512 --height 512 --steps 10 --open
/i2i --image ./input.png "make it watercolor" --steps 20 --guidance-scale 6.5 --open

For a newer but still relatively small local model, try black-forest-labs/FLUX.2-klein-4B after installing Diffusers from source (see docs/getting-started.md):

/backend diffusers black-forest-labs/FLUX.2-klein-4B mps float16
/t2i "a product photo of a matte black espresso machine" --steps 4 --guidance-scale 1.0 --open

Local Diffusers text_to_video remains experimental and is temporarily disabled from the normal bundled local surfaces. Use MLX-Gen Wan 2.2 TI2V for local Apple Silicon text_to_video / image_to_video, or use the OpenAI-compatible backend for remote video endpoints.

For Apple Silicon local generation through MLX-Gen:

/backend mlx-gen AbstractFramework/flux.2-klein-4b-4bit
/t2i "a product photo of a matte black espresso machine" --steps 4 --guidance-scale 1.0 --open
/backend mlx-gen AbstractFramework/qwen-image-edit-2511-4bit
/i2i --image ./input.png "replace the background with a clean white studio setup" --steps 20 --guidance-scale 2.5 --strength 0.75 --open
/backend mlx-gen Wan-AI/Wan2.2-TI2V-5B-Diffusers
/t2v "a red fox walking through a snowy forest, cinematic" --frames 121 --fps 24 --steps 50 --guidance-scale 5.0 --open
/i2v --image ./first-frame.png "slow camera push-in" --frames 121 --fps 24 --steps 50 --guidance-scale 5.0 --open

/t2v and /i2v show MLX-Gen video progress by default; add --no-progress to suppress progress output in scripts.

OpenAI-compatible server example:

/backend openai http://localhost:1234/v1
/t2i "a watercolor painting of a lighthouse" --width 512 --height 512 --steps 10 --open

The CLI/REPL can also be configured via ABSTRACTVISION_* env vars; see docs/reference/configuration.md.

Local web playground

The playground is owned by AbstractVision and runs without AbstractCore. It is a local/dev testing surface; use AbstractCore/Gateway for production routing, authentication, and browser-origin policy.

abstractvision playground --port 8091

Open http://127.0.0.1:8091/vision_playground.html. The page and the API are served by the same process.

Current behavior:

• The UI is split into task tabs (Text→Image, Image→Image, Text→Video, and a placeholder Image→Video tab for later work).
• Each active task tab has its own model selector and unload button. Switching models in a tab unloads the current active backend first to free memory before loading the replacement.
• The Image→Image tab is enabled only for models that both advertise image_to_image in the packaged capability registry and remain enabled by the selected backend.
• MLX-Gen FLUX.2 klein/base, Qwen Image Edit, ERNIE Image Turbo, FIBO, and FIBO Edit models are surfaced for Image→Image edits when cached.
• The Text→Video tab is experimental in the playground UI. Use shell/REPL or AbstractCore for the most complete MLX-Gen Wan video controls.
• Model-specific request normalization happens at the API/backend layer, not just in the page.
• Local video export packages generated frames into MP4 via an external ffmpeg binary on PATH.
• Response logs intentionally show only a shortened b64_json preview instead of the full base64 image payload.

One-shot commands default to the OpenAI-compatible HTTP backend, but they also support local providers:

abstractvision t2i --base-url http://localhost:1234/v1 "a studio photo of an espresso machine"
abstractvision i2i --base-url http://localhost:1234/v1 --image ./input.png "make it watercolor"
abstractvision t2i --provider diffusers --model qwen-image "a studio photo of an espresso machine"

Local GGUF via stable-diffusion.cpp

If you want to run GGUF diffusion models locally, use the stable-diffusion.cpp backend (sdcpp). Start with a single-file Stable Diffusion model when possible; Qwen Image and FLUX GGUF component sets are heavier.

Recommended:

• abstractvision auto-installs sd-cli into ~/.abstractvision/bin on first use (set ABSTRACTVISION_SDCPP_AUTO_INSTALL=0 to disable).
• If you prefer python bindings: install abstractvision[sdcpp] (uses stable-diffusion-cpp-python).

Alternative (external executable): install sd-cli from https://github.com/leejet/stable-diffusion.cpp/releases.

In the REPL:

/backend sdcpp /path/to/sd-v1-5.gguf /path/to/sd-cli
/t2i "a watercolor painting of a lighthouse" --width 512 --height 512 --steps 10 --open

Curated FLUX/Qwen GGUF bundle example:

abstractvision download flux2-klein-base-4b --provider sdcpp
abstractvision download qwen-image-edit-2511-gguf --provider sdcpp

/backend sdcpp flux2-klein-base-4b /path/to/sd-cli
/t2i "a product photo of a matte black espresso machine" --steps 4 --guidance-scale 1.0 --sampling-method euler --diffusion-fa --offload-to-cpu --open

The package resolves the required VAE and text-encoder companions from the cache automatically for curated sdcpp model keys. Manual component wiring remains available for advanced cases.

Extra flags are forwarded via request.extra. In CLI mode they are forwarded to sd-cli; in python bindings mode, keys are mapped to python binding kwargs when supported and unsupported keys are ignored.

AbstractCore tool integration (artifact refs)

If you’re using AbstractCore tool calling, AbstractVision can expose vision tasks as tools:

from abstractvision.integrations.abstractcore import make_vision_tools

tools = make_vision_tools(vision_manager=vm, model_id="Qwen/Qwen-Image-Edit-2511")

Install abstractcore in the host application environment when you use these helpers; it is not pulled in by AbstractVision.

AbstractFramework ecosystem

AbstractVision is part of the AbstractFramework ecosystem and is designed to compose with:

• AbstractFramework (project hub): https://github.com/lpalbou/AbstractFramework
• AbstractCore (orchestration + tool calling): https://github.com/lpalbou/abstractcore
• AbstractRuntime (runtime services, including artifact storage): https://github.com/lpalbou/abstractruntime

In practice:

• AbstractVision standardizes generative vision outputs (image/video) behind VisionManager.
• AbstractCore can discover and use AbstractVision via the capability plugin (src/abstractvision/integrations/abstractcore_plugin.py) or you can expose vision tasks as tools (src/abstractvision/integrations/abstractcore.py).
• Artifact refs returned by AbstractVision are designed to travel across processes; RuntimeArtifactStoreAdapter bridges to an AbstractRuntime-style artifact store (src/abstractvision/artifacts.py).

Project

• Release notes: CHANGELOG.md
• Contributing: CONTRIBUTING.md
• Security: SECURITY.md
• Acknowledgments: ACKNOWLEDGMENTS.md
• Agent docs: llms.txt and llms-full.txt

Requirements

• Python >= 3.9

License

MIT License - see LICENSE file for details.

Author

Laurent-Philippe Albou

Contact

[e-mail]