LocalVLM Image Caption (GGUF + mmproj)

Image in, text out, using a multimodal GGUF on your own GPU. Two jobs, and they're both common enough that you probably want one: captioning a training set, and seeding a prompt from an existing image for img2img or image-to-video.

The training-set job is where local earns its keep. The whole reason JoyCaption exists is that the alternatives were "an expensive censored API or a weak open model" - uncensored, equal SFW/NSFW coverage, offline, free per image. Same logic here: an abliterated VLM won't hand you "cylindrical object" where a body part should be. And if you're captioning for a Krea 2 workflow, note what Krea 2 encodes with - Qwen3-VL, chosen by Krea so the same stack could later support editing. A natural-language caption is close to native input for it, which is why the pack ships a Krea2/FLUX2 caption template rather than a tagger.

How it works

A VLM needs two files: the quantized model and the mmproj, the vision projector that lets the language model see images. Set mmproj to auto and the node uses the projector belonging to the catalog model, downloading it if needed (~1 GB for the Mistral pair). Local files need the matching mmproj-*.gguf from the same HuggingFace repo, picked by name in the dropdown.

Each image becomes a base64 PNG data URI, downscaled to a max of 1568 px on the long side with Lanczos before encoding - image tokens are expensive and a 4K still buys you nothing but latency. Then one chat request per batch item with the image first and your prompt second, and the results are joined with batch_separator (default \n---\n; the widget really does show a literal backslash-n, and the code converts it).

Vision also needs a chat handler from your llama-cpp-python build - the node walks Qwen3VL → Qwen2.5VL → MiniCPM-V → Llava16 → Llava15, nudged toward the right family by the model's filename (Mistral/Pixtral go down the Llava path). If none exists, you get No compatible multimodal chat handler found, which is not a bug in the node - your llama.cpp is too old for the architecture.

The inputs that matter

• model - only vision-capable families are listed, so you can't accidentally pick the text-only Qwen3-4B. Mistral-Small-3.2-24B abliterated is the vision family the pack is built around.
• mmproj - auto for catalog models. Wrong or missing projector is the single most common mistake: the model loads, ignores the image, and invents a scene.
• image - a batch works; each frame is captioned separately and joined, so four images come back as one string with separators.
• prompt - the instruction. The default asks for subject, setting, style, lighting, colours and composition.
• system_prompt - defaults to "You are a precise image captioning assistant." A connected system_prompt_input overrides it.
• temperature 0.4, max_tokens 2048, n_ctx 8192 - captioning wants a low temperature, and 1568 px of image eats a big slurp of that context.
• n_gpu_layers -1, batch_separator, unload_after_run.

Output caption → Show Text or Save Text for a dataset, into the Prompt Generator's text_input for a rewrite, or straight into CLIP Text Encode.

Chaining it (the workflow people actually build)

[Load Image] -> [LocalVLM Image Caption] -> [LocalLLM Prompt Generator (text_input)] -> [CLIP Text Encode]

Caption first, then let the text model reshape it for your base. Both nodes share one cached model, so the whole chain loads the LLM once - as long as model, mmproj, n_ctx and n_gpu_layers stay identical. Change any of those and you eat a reload, which for a 24B is tens of seconds.

Install and the usual fires

Same as the rest of the pack: ComfyUI Manager → search Slarti LLM Nodes, or clone https://github.com/Slartibart23/ComfyUI-Slarti-LLM-Nodes into ComfyUI/custom_nodes and restart. requirements.txt deliberately omits llama-cpp-python - a plain pip install is CPU-only - so the pack installs a CUDA build itself on first run. Restart afterwards and check the startup line says GPU. Set LOCALLLM_NO_AUTOINSTALL=1 if you manage your own environment.

Where it breaks:

• Model loads, image ignored. Mismatched mmproj. Use the projector from the same repo as the model, not a lookalike.
• 'X' is a text-only model and cannot see images. You picked a text-only catalog entry with auto mmproj. Untick that; pick a vision family.
• Empty dropdowns / No GGUF model found. Files aren't in ComfyUI/models/LLM/, or ComfyUI wasn't restarted after you copied them.
• Out of memory. The VRAM check adds ~1 GB for the projector on top of the model, so a 14.4 GB quant is really ~15.4 GB plus context. Lower n_gpu_layers, drop n_ctx, or use a smaller quant.
• Batch behaviour surprise. You get one joined string, not a caption per image. For a dataset, feed one image per run - or a filename-aware loader and a save step that keeps the image name - otherwise you'll lose track of which caption goes with which picture.

One honest caveat that applies to every VLM in this class: multi-subject attribution is where they fail. Two people in frame and the model will happily hand the red dress to the wrong one. Auto-caption the hundred-image set, hand-audit the twenty that matter.