cgh-vision

Image understanding for cgh: a content inventory decides what each image contains, then only the extractors the content warrants run: architecture diagrams become markdown plus Mermaid, tables and charts become data, dense text becomes a summary. Everything runs against a local Ollama daemon; nothing leaves the machine.

pip install cgh-vision
ollama pull qwen2.5vl:3b gemma3:4b   # the benchmark-selected pair

cgh vision docs/architecture.png                 # markdown on stdout
cgh vision photo.jpg --profile photo             # screen-photo tuning
cgh vision archi.png --out report.md             # also save the report
cgh vision archi.png --format json | jq .diagram # the SDK dicts instead

The command shows a progress spinner on stderr while the model passes run (about 30 s per diagram on Apple Silicon); stdout stays pure markdown or JSON, pipeable either way.

Without Ollama, one command: cgh vision setup --llamacpp

If Ollama is unavailable and you have llama.cpp (or can install it), this wires everything up:

cgh vision setup --llamacpp

It finds llama-server (offering brew install llama.cpp on macOS, or the signed GitHub-release binaries on Windows, official channels only), writes a [plugin.vision] block pointing cgh-vision at a local llama-server, and offers to start it. The server pulls our default vision model (ggml-org/Qwen2.5-VL-3B-Instruct-GGUF, weights and mmproj projector) itself on first run. The server is yours to keep running and to stop, exactly like the Ollama daemon; cgh starts it on request but does not supervise it.

Benchmark: llama-server gave the best node and edge scores of any transport (see the plugin's own notes). This is the recommended no-Ollama path.

Without Ollama: any OpenAI-compatible endpoint

Ollama is the default, not a requirement. Setting openai_base_url switches the transport to /chat/completions with a base64 image_url, which unlocks three things:

• No daemon at all. Serve the GGUF weights you downloaded from Hugging Face with llama.cpp's own server, which is what Ollama wraps anyway:

  llama-server -m qwen2.5-vl-7b-q4_k_m.gguf \
    --mmproj qwen2.5-vl-7b-mmproj-f16.gguf --port 8080
  

  [plugin.vision]
  openai_base_url = "http://127.0.0.1:8080/v1"
  nodes_model = "qwen2.5-vl"   # whatever name the server reports
  edges_model = "qwen2.5-vl"
  fallback_model = ""
  

  A loopback endpoint stays "local", so secure mode is satisfied and nothing leaves the machine.

• LM Studio, vLLM, or an approved internal gateway, same config, just a different openai_base_url. A key is read from the env var named by openai_api_key_env (default OPENAI_API_KEY).

  Node and edge extraction over the OpenAI transport is on par with native Ollama; zone detection is a little weaker, because the /chat/completions template differs from the raw prompt Ollama's native API takes. If zones matter most, keep the Ollama backend.

• Hosted vision models (a corporate LLM gateway serving qwen-vl, GLM-4V, and such). These are non-loopback, so cgh treats them as cloud: allowed in assist mode with an audit line, refused in secure mode, exactly like a remote Ollama.

Installing Ollama

cgh vision needs the Ollama daemon. It does not bundle it: when the daemon is unreachable, cgh points at the publisher's official channel for your OS and, interactively, offers to run it.

winget is chosen on Windows because managed machines commonly allow it where a raw .exe download is blocked. If your network blocks every official channel, that is usually a deliberate policy: ask IT to whitelist Ollama, or point cgh at an approved internal Ollama server with ollama_url. cgh will not mirror or obfuscate the installer to get it past a content filter.

When ollama pull is blocked

Corporate machines often block the Ollama registry (or the installer itself). Check first, in one command:

ollama pull qwen2.5vl:3b   # works? nothing else to read here

If it fails, the models can come from Hugging Face instead and be registered locally. The vision models need two files: the weights and the vision projector, which is what makes the model multimodal; without it Ollama loads a text-only model and every image is ignored.

pip install -U "huggingface_hub[cli]"
mkdir -p ~/models/qwen2.5-vl-7b && cd ~/models/qwen2.5-vl-7b

# weights (q4_k_m is about 5 GB, q8_0 about 8 GB for more quality)
hf download Mungert/Qwen2.5-VL-7B-Instruct-GGUF \
  Qwen2.5-VL-7B-Instruct-q4_k_m.gguf --local-dir .

# the vision projector, mandatory
hf download Mungert/Qwen2.5-VL-7B-Instruct-GGUF \
  Qwen2.5-VL-7B-Instruct-mmproj-f16.gguf --local-dir .

cat > Modelfile <<'EOF'
FROM ./Qwen2.5-VL-7B-Instruct-q4_k_m.gguf
FROM ./Qwen2.5-VL-7B-Instruct-mmproj-f16.gguf
EOF

ollama create qwen2.5-vl:7b -f Modelfile

Then point the plugin at the name you registered, since it will not match the registry tag:

[plugin.vision]
nodes_model = "qwen2.5-vl:7b"
edges_model = "qwen2.5-vl:7b"   # or another locally created model
fallback_model = ""             # unless you registered a second one

cgh vision --help and the daemon probe behave identically: cgh only ever asks Ollama for a model name, it never downloads anything itself.

Pipeline

1. Inventory (non-directive: an image is never assumed to be a diagram; a logo or a photo costs one call and one summary line).
2. Pre-scaling for small images: below 1000 px (smaller dimension), a 2x Lanczos upscale feeds the diagram passes, which the benchmark showed rescues thin-line exports (drawio) without ever hurting the others.
3. Diagram extraction when warranted: structure with the plain contract, enrichment over the found labels (title, kinds, technologies, legend only if actually drawn), then a second model reads the arrows constrained to the found labels. Benchmarked ensemble: node precision 1.00, edge recall 0.80.
4. Fallback reader when the structure comes back skeletal (two boxes or fewer, or no arrows at all): the arrow model reads the structure a second time and wins only if it finds more. The two models fail differently, which is what makes the retry worth anything: benchmarked over every local vision model, gemma3:4b rescues all five thin-line cases the primary reader cannot see (2 nodes / 1 edge becoming 13 / 27), needs no extra download, and never fires on images already read correctly.
5. Table / chart / text extractors as routed.
6. Post-processing: fuzzy-duplicate merge, arrow annotations dropped from node lists, reversed-edge dedup, and identity separation: IPs, CIDRs, FQDNs, emails and server names split out of labels, recorded as pii.image_identity findings so the secure-at-rest layer pseudonymizes them.

Findings

Configuration

[plugin.vision]
# profile = "default"        # default | fast (single call) | photo
# nodes_model = "qwen2.5vl:3b"
# edges_model = "gemma3:4b"  # set to "" to disable the edge pass
# ollama_url = "http://127.0.0.1:11434"
# timeout_s = 120
# fallback_model = "gemma3:4b"     # second reader, "" disables it
# prescale = true            # 2x upscale of small images (see Pipeline)
# prescale_min_px = 1000     # apply below this smaller-dimension size
# min_bytes = 5120           # skip icons and badges
# max_bytes = 20971520

Embedding

The pipeline is part of the cgh SDK surface (codegraph.sdk.image_*, MIT): image_inventory, extract_diagram, extract_table, extract_chart. See cgh's docs/EMBEDDING.md.