argus-lens 0.2.0

Intent-aware, multi-model captioning pipeline for LoRA training, dataset curation, and generative AI workflows

Argus Lens

One image. A hundred perspectives.

Multi-model captioning pipeline for LoRA training, dataset curation, and generative AI workflows. Unlike traditional captioners that describe an image, Argus Lens produces intent-aware caption subsets -- structured, filtered, and optimised for how captions are actually used downstream.

Looking for the web UI? See argus-vision-demo -- a thin Next.js frontend for exploring argus-lens interactively.

Quick Start

pip install argus-lens[openai]

from argus_lens import ArgusLens

engine = ArgusLens(backend="openai", api_key="sk-...")
result = engine.caption("photo.jpg", trigger_word="sks_person")

print(result.final_caption)
print(result.caption_variants["training"])
print(result.caption_variants["zeroshot"])

Why Argus Lens?

Most captioning tools answer "what's in this image?" -- Argus Lens answers "how will this caption be used?"

BLIP gives you a sentence. WD14 gives you a flat tag list. Neither knows whether you're training a LoRA, curating a dataset, or generating zero-shot. Argus Lens produces multiple structured caption variants from the same image, each optimised for a different downstream task.

	Typical captioner	Argus Lens
Output	Single flat string	Structured, category-bucketed variants
Intent awareness	None -- describes the image	Training, zero-shot, and per-category variants
Multi-model fusion	One model at a time	Hybrid pipelines (e.g. WD14 tags + GPT-4o prose)
Token budget management	Manual / none	Auto-tuned for SDXL (CLIP, 60 tokens) vs Flux/SD3 (T5, 200 tokens)
LoRA training support	Raw output, hope for the best	Identity suppression, omission cycles, tiered tag protection
Dataset workflow focus	Captioning only	Batch processing, deduplication, export to txt/JSON/JSONL/CSV
Transparency	Black box	Every removed phrase and compaction decision is reported

What the assembly pipeline does

The model is just an input source. The real value is what happens after inference:

1. Classify -- every tag and prose clause is bucketed into identity, wardrobe, pose/composition, setting, lighting, or action
2. Filter -- redundant prose (>50% overlap with tags), filler prefixes ("The image shows..."), and training noise (rating tags, meta tokens) are stripped
3. Specialise -- the training variant suppresses identity traits (the LoRA learns these visually; stating them conflicts with the base model's prior), while the zero-shot variant puts identity first (no LoRA to supply it)
4. Budget -- fragments are assembled under the correct CLIP/T5 token limit with tiered protection (framing tags are never dropped, short generic tags are shed first)
5. Diversify -- omission cycles systematically suppress different category buckets across images, creating stronger concept disentanglement across the dataset
6. Enrich -- novel compound nouns from prose output ("gray sweater", "wooden door") are extracted and appended at lowest priority to fill remaining token budget

Features

• Multi-model backends: WD14, Florence-2 (local GPU/CPU) + OpenAI, HuggingFace, Replicate, NVIDIA NIM (cloud API)
• Structured captions: Category-bucketed variants (identity, wardrobe, pose, setting, lighting, action)
• Training-optimised: Tiered tag protection, omission cycles, CLIP/T5 token budgets, identity suppression
• Zero-shot variant: Identity-first, prose-preferred captions for generation without LoRA
• Hybrid pipelines: Mix local + cloud backends (e.g. WD14 tags + GPT-4o prose)
• Backend-aware budgets: Automatic token limits for SDXL (60), Flux (200), SD3 (200)
• Deterministic and filterable: Category-aware outputs you can audit, filter, and override
• CLI + Server: Command-line tool and optional FastAPI micro-server
• Export formats: .txt sidecars, JSON, JSONL, CSV

Installation

pip handles all Python dependencies through extras. Pick the extras that match your use case:

# Assembly engine only (no model deps)
pip install argus-lens

# Local backends (GPU inference)
pip install argus-lens[local]      # WD14 + Florence-2
pip install argus-lens[wd14]       # WD14 only (CPU, no torch)
pip install argus-lens[torch]      # Florence-2 only

# Cloud backends (no GPU needed)
pip install argus-lens[openai]     # GPT-4o vision
pip install argus-lens[replicate]  # Replicate API

# Server (FastAPI + uvicorn)
pip install argus-lens[server,local,openai]

# Everything
pip install argus-lens[all]

If you're adding argus-lens to an existing project, just add e.g. argus-lens[openai] to your requirements.txt -- pip resolves all transitive deps automatically.

System dependencies for local GPU backends

Cloud-only users ([openai], [replicate]) need no system packages -- skip this section.

Local backends ([local], [wd14], [torch]) require system libraries for image processing and (optionally) CUDA for GPU acceleration. On Ubuntu/Debian:

sudo apt install -y \
    libgl1 libglib2.0-0 libxcb1 libsm6 libxext6 libxrender1

For GPU inference, you also need:

• NVIDIA GPU drivers (check with nvidia-smi)
• CUDA runtime (the Dockerfile.gpu-base in this repo uses nvidia/cuda:12.4.1-runtime-ubuntu22.04 as a reference)
• NVIDIA Container Toolkit (for Docker deployment only)

If you already have torch and CUDA working in your environment, you're set -- the pip extras handle the rest.

Usage

Python API

Import and use directly in your code. This is the primary interface.

from argus_lens import ArgusLens

# Cloud backend -- works anywhere, no GPU
engine = ArgusLens(backend="openai", api_key="sk-...")
result = engine.caption("photo.jpg", trigger_word="sks_person")

# Local backend -- needs torch + GPU/CPU
engine = ArgusLens(backend="hybrid")
result = engine.caption("photo.jpg", trigger_word="sks_person")

# Batch processing
results = engine.caption_directory("./images/", output_format="txt")

CLI

# Caption a single image
argus-lens caption photo.jpg --trigger sks_person --backend openai

# Caption a directory, output as txt sidecars
argus-lens caption ./images/ --format txt --backend hybrid

# List available backends
argus-lens backends

HTTP Server

Run the built-in FastAPI server for frontend consumers (e.g. argus-vision-demo):

pip install argus-lens[server,local]
argus-lens serve --cors --port 8080

Endpoints:

• POST /caption -- multipart file upload
• POST /caption/url -- JSON body with image URL
• POST /caption/batch -- multiple file upload
• POST /caption/stream -- NDJSON streaming for batch
• GET /backends -- list available backends

Docker

For fresh hosts or isolated deployment with GPU passthrough. No pip install needed on the host.

# Build and run
./build-docker.sh
docker compose up

This builds a CUDA 12.4 base image, installs all extras into it, and runs argus-lens serve on port 8080.

Configuration

Copy or create a .env file for the Docker deployment:

Variable	Default	Description
ARGUS_BACKEND	hybrid	Captioning backend (hybrid, wd14, florence2, openai, etc.)
ARGUS_API_KEY	--	API key for cloud backends
ARGUS_PORT	8080	Host port for the server
WD14_MODEL_DIR	~/.cache/wd14_tagger/	WD14 ONNX model directory (auto-downloads on first use)
HF_HOME	~/.cache/huggingface	HuggingFace model cache (auto-downloads on first use)
HF_TRUST_REMOTE_CODE	false	Only needed for legacy microsoft/Florence-2-* weights. See Security

GPU prerequisites

# Verify NVIDIA driver
nvidia-smi

# Install container toolkit (if not already)
sudo apt install nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Model caching

The docker-compose.yaml bind-mounts ~/.cache/wd14_tagger and ~/.cache/huggingface from the host so models persist across container rebuilds. Models auto-download on first use if not already cached.

Security

trust_remote_code and Florence-2

By default, the Florence-2 backend uses florence-community/Florence-2-base weights which are natively supported in transformers -- no trust_remote_code needed.

The legacy microsoft/Florence-2-base weights require HF_TRUST_REMOTE_CODE=true, which executes arbitrary Python from the model repository at load time. Only enable this for models you trust. WD14 uses a static ONNX model and never runs remote code.

License

MIT