🔍 llmog (LLM Object Detection)

A library for object detection tasks using LLMs.

This project implements an iterative Detector-Judge pipeline: a VLM "detector" agent proposes bounding boxes, a VLM "judge" agent critiques them against the original image, and the loop repeats with structured feedback until the annotations meet a quality threshold or the round limit is reached.

Table of Contents

• What This Project Can Do For You
• Key Features
• Quick Start
• Try It Online
• Coding Snippets
  • 1. Free-Detection CLI
  • 2. Auto-Labeling CLI
  • 3. Whole-Image Classification CLI
  • 4. Launching the Web GUI
  • 5. YAML Config + CLI Overrides
• Output Structure
• Technology Stack

🎯 What This Project Can Do For You

Stop hand-labeling and let a vision-language model do the heavy lifting. llmog turns any OpenAI-compatible VLM (llama.cpp, vLLM, Ollama, cloud APIs) into a practical object-detection tool for real workflows:

• Detect anything, zero training — give it a comma-separated category list ("person, car, dog" or "crack, scratch, dent") and it detects those objects in any image, with no fine-tuning, no dataset, and no YOLO weights required.
• Produce high-quality annotations, automatically — the built-in judge agent critiques every round of detections and the loop self-corrects, so the final boxes are far more reliable than a single naive VLM pass.
• Relabel your existing YOLO datasets — have a binary defect/no-defect dataset? auto_label walks every box, asks the VLM to classify each crop into a multi-class label, and writes back an updated data.yaml — turning 2-class datasets into rich multi-class ones.
• Scale to small, hard-to-see objects — tiling, CLAHE contrast, denoising, sharpening, Set-of-Mark prompting, and crop-verify passes help the VLM find tiny defects that would otherwise be missed.
• Run on your own hardware — spin up and manage local llama-server or vLLM instances directly from the CLI or the web UI; no cloud keys required.
• Interact visually — launch a Gradio web console to batch-test images, watch live per-round annotated results, tweak preprocessing, and download a .zip of everything.

Common use cases

🌟 Key Features

Core pipeline

• Iterative Refinement Loop — enhances detection accuracy by feeding visual and text-based critiques back to the detector across multiple rounds.
• Unified CLI — a single entry point (uv run llmog --task ...) routes to either workflow; the legacy detection-cli and auto-annotation aliases are kept as shortcuts.
• YAML Config + CLI Precedence — every flag can live in a YAML file loaded via --config. Explicit CLI flags always win over YAML, which wins over framework defaults.
• Batch YOLO Auto-Labeling — auto_label mode walks an existing defect/no-defect dataset, asks the VLM to classify each cropped box, accumulates a class_map, and updates data.yaml in place — with resumable checkpoints and batched I/O.
• Visual Annotations & Grids — automatically overlays a customizable 0–1000 coordinate grid on images to aid the VLM's spatial reasoning, and draws lime-green bounding boxes with legible text labels for final detections.
• Robust File Persistence — saves the best annotated image, raw JSON detections, and complete round-by-round history for every processed image.

Advanced image preprocessing & augmentation

• Dynamic Resolution Tuning — custom upscaling target for the short edge, with optional letterbox padding to preserve aspect ratio on square inputs.
• Contrast & Color Enhancements — LAB color-space CLAHE, global autocontrast, gamma correction (0.5–2.0), and Gray World white balance correction.
• Noise Filtering & Sharpening — bilateral or Non-Local Means (NLM) denoising with edge preservation, plus unsharp mask sharpening.
• Tiling Engine for Small Objects — slices high-resolution inputs into overlapping sub-patches, runs sequential detection on each tile, and merges overlapping detections with Non-Maximum Suppression (NMS).
• Crop & Verify Validation — runs a second-pass confirmation on cropped candidate coordinates with context padding to reduce hallucinated detections.
• Set-of-Mark (SoM) Prompting — classic CV contour detection overlays numbered candidate regions onto the image, turning the spatial regression task into a simpler classification/selection task.
• Fully Customizable Grid Overlays — adjustable style (standard, transparent, fine, none), step size, line thickness, font size, and custom colors (line, text, and text-backing box — CSS name or hex).
• VLM Processor Pixel Bounds — manually configure min_pixels and max_pixels request parameters (passed via the API request's extra_body) to tune vision encoder resolution and prevent OOMs on backends like vLLM / Qwen-VL.

Server lifecycle managers (llmog/servers/)

• 🦙 LlamaServerManager — manage the native llama-server binary: ports, context sizes (--ctx-size), KV cache quantization (--cache-type-k/--cache-type-v), parallel request slots, draft models, flash attention, and reasoning toggles.
• 🐍 LlamaCppPythonManager — the same API surface driven by llama-cpp-python's bundled server (python -m llama_cpp.server); pick it with --server_type llama_cpp_python and pip install -e ".[llama-cpp-python]".
• ⚡ VllmServerManager — launch and control local vLLM instances with tensor/pipeline parallelism, custom dtypes, prefix caching, and chunked prefill options.

Gradio web interface

• 🦙 Server Management — start, stop, and monitor multiple local llama-server / vLLM instances directly within the GUI.
• ⚙️ Detection Settings — customize target categories, category definitions, prompt templates, per-role model/server routing, and sampling parameters.
• 🧪 Batch Test — upload multiple images, run them sequentially, view live per-round annotated results, and download all outputs as a .zip archive.
• 🖼️ Interactive Preprocessing & Custom Grids — tweak all resolution scaling, filters, tiling options, grid colors, text sizes, and pixel bounds directly through visual controls.

🚀 Quick Start

Requirements: Python 3.12+, uv, and (optionally) a CUDA-capable GPU for local inference.

git clone https://github.com/mohamed-em2m/llm-object-grounding.git
cd llm-object-grounding

uv sync

Install one serving backend:

uv pip install -e ".[llama-cpp-python]"    # Python binding, ships its own server (--server_type llama_cpp_python)
uv pip install -e ".[llama-cpp]"           # native llama.cpp binary (default; build with ./scripts/install_llama_cpp.sh)
uv pip install -e ".[vllm]"                # vLLM on CUDA (--server_type vllm)

🚀 Try It Online

No local setup required. Run the auto-annotation pipeline directly in the cloud:

💡 Tip: Google Colab is recommended for a quick interactive demo, while Kaggle provides another convenient environment for running the notebook with GPU acceleration.

🖥️ Coding Snippets

There is one unified entry point — llmog — that dispatches by --task. Both workflows also have shortcut aliases that pre-set the right --task for you.

Single source of truth: every CLI flag is defined on llmog/schemes/argument.py:PipelineConfig (a pydantic v2 model). llmog/main.py:build_parser mirrors that model onto an argparse.ArgumentParser, and parse_args() overlays an optional --config <yaml> file before constructing the validated PipelineConfig. Both dashed (--prep-tile-size) and underscored (--prep_tile_size) forms are accepted.

1. Free-Detection CLI

Run the detector→judge pipeline on one or more images:

uv run llmog --task free_detection -i image.jpg -c "person, car, dog"

# Shortcut alias
uv run detection-cli -i image.jpg -c "person, car, dog"

Common options

Preprocessing pipeline options

Custom grid & VLM pixel bounds options

Example running CLAHE, NMS tiling, a custom blue grid, and custom Qwen-VL pixel bounds:

uv run llmog --task free_detection \
  -i industrial_part.jpg \
  -c "crack, scratch, dent" \
  --prep_enabled \
  --prep_contrast_method clahe \
  --prep_tiling_enabled \
  --prep_tile_size 512 \
  --prep_tile_overlap 0.25 \
  --prep_grid_line_color "blue" \
  --prep_grid_step 50 \
  --prep_send_pixel_bounds \
  --prep_min_pixels 200704 \
  --prep_max_pixels 2097152 \
  -o ./inspection_results

2. Auto-Labeling CLI (--task auto_label)

Walk an existing YOLO-format dataset (binary defect/no-defect), re-classify each annotated crop with a VLM, accumulate a class map, and persist the updated data.yaml.

uv run llmog --task auto_label \
    --train_image imgs/ --train_label lbls/ \
    --yaml_path data.yaml \
    --model local-model \
    -o ./out

# Shortcut alias
uv run auto-annotation --train_image imgs/ --train_label lbls/ \
    --yaml_path data.yaml --model local-model -o ./out

Full reference: uv run llmog --help (every flag mirrors a PipelineConfig field in llmog/schemes/argument.py).

Auto-label-specific options

Crop size + empty-prediction example:

# 640px square crops; "none"-like crops produce empty .txt files (YOLO "no objects")
uv run llmog --task auto_label \
    --train_image imgs/ --train_label lbls/ \
    --yaml_path data.yaml --model local-model -o ./out \
    --image_size 640

# Custom none vocabulary + keep legacy behavior (none becomes a real class)
uv run llmog --task auto_label \
    --train_image imgs/ --train_label lbls/ \
    --yaml_path data.yaml --model local-model -o ./out \
    --none_labels "none,empty,background" --keep_none

3. Whole-Image Classification CLI (--task classify)

Classify full images (no boxes) into single, multi, or top_k predictions, with CSV and/or YOLO-cls outputs. --class_mode works exactly as in auto-label: strict locks to --categories (else none), hybrid may discover new classes, free names whatever it sees.

uv run llmog --task classify -i img1.jpg -i img2.jpg \
    -c "hole, stain, tear" --classification_mode single -o ./cls_out

uv run llmog --task classify --input_folder imgs/ \
    -c "car, truck, pedestrian" --classification_mode top_k --top_k 3 \
    --output_format both -o ./cls_out

4. Launching the Web GUI

To launch the interactive Gradio interface:

uv run detection-gui

Options:

Example:

uv run detection-gui --port 7861 --share

5. YAML Config + CLI Overrides

Any field on PipelineConfig can live in a YAML file loaded by --config. A fully-commented example is at examples/config.example.yaml.

# pipeline.yaml
task: free_detection
categories: "crack, scratch, dent"
max_rounds: 5
score_threshold: 9
output_folder: ./yaml_out
prep_enabled: true
prep_contrast_method: clahe
prep_tiling_enabled: true

# YAML provides defaults; CLI flags override only what they explicitly set.
uv run llmog --task free_detection --config pipeline.yaml -i img.jpg --max_rounds 2
# → max_rounds == 2             (CLI won)
# → score_threshold == 9        (from YAML)
# → output_folder == ./yaml_out (from YAML)
# → prep_enabled == true        (from YAML)

# Full example: detect on the included sample config
uv run llmog --config examples/config.example.yaml

📁 Output Structure

free_detection task

For every image processed, a dedicated subdirectory is created under --output_folder:

detection_results/
├── summary.json             # Per-image status, best round + score
└── [image_name]/
    ├── best_annotated.jpg    # Best annotated image with lime-green bounding boxes
    ├── best_detections.json  # Final parsed JSON detections
    └── history.json          # Complete round-by-round scores, detections, and feedback

auto_label task

Relabeled .txt files (YOLO format with the discovered class IDs) plus a checkpoint and an updated data.yaml:

out/
├── labels/                 # (when not using --inplace_saving)
│   ├── img_0001.txt
│   └── ...
├── .checkpoint.json        # Auto-resume state (only if --auto_resume is on)
└── data.yaml                # Updated copy with the discovered class_map

🛠️ Technology Stack

• Core Logic: Python 3.12+, Pillow (PIL) for image manipulation, OpenCV (opencv-python) for CLAHE/contour/bilateral processing, Matplotlib for rendering.
• VLM Integrations: OpenAI Python SDK, compatible with any OpenAI-style endpoint (llama-server, vLLM, Ollama, etc.), with support for custom payloads via extra_body.
• Web UI: Gradio.
• Environment & Package Management: uv, Setuptools.

Repository: mohamed-em2m/llm-object-grounding — a framework for testing LLMs on object grounding. Docs: github.com/mohamed-em2m/llm-object-grounding/tree/main/docs