LLaVA Captioner 🌊

LLaVA Captioner 🌊 is the one node in the ceruleandeep/ComfyUI-LLaVA-Captioner pack, and it does exactly one thing: lets you ask an image questions in plain English and get a text answer back, entirely on your own machine. No API key, no cloud service, no filter, nothing leaves your box. You feed it an image and a prompt like "describe this in 10 to 20 words" or "how many people are in this picture?", and it returns a caption or an answer.

The name undersells it - it's less a captioner and more a local vision chatbot. The README is blunt about capability: the underlying model is "maybe as smart as GPT3.5, and it can see." That's the honest framing. This won't beat JoyCaption or a WD14 tagger for bulk LoRA training captions (those are the modern favorites for a reason), but it's the node you reach for when you want to ask something about an image without leaving ComfyUI or paying per call.

How it works

Under the hood it runs a GGUF-quantized LLaVA 1.5 model through llama-cpp-python. The node base64-encodes your image into a PNG data URL, stuffs it into a chat completion message alongside your prompt, and lets the model answer. It also injects a hardcoded system message that pushes for factual, literal descriptions of what's actually in the frame - no metaphors, no interpretation. The author's framing ("Describe only what you see, not what you think the image is about") is built in, which is why the output reads so flat and trustworthy for tagging.

Mechanically worth knowing: it loads the full model once per run, then processes every image in your batch with a progress bar, joining the results with newlines. The context window is 2048 tokens because multimodal LLaVA needs roughly 4x the context of text-only Llama - that's the main reason CPU inference crawls at about 25 seconds per image. On a GPU it's more like 4 seconds.

The inputs that matter

Six inputs, all required, and only three you'll actually fiddle with:

• image - the IMAGE tensor you're asking about. Batches work; each one gets captioned and results are joined by newlines.
• prompt - the question, in natural language. Default is "Please describe this image in 10 to 20 words." Change it to get tags, counts, or adversarial descriptions.
• max_tokens - response length cap (default 40, 0–200). A token is roughly half a word.
• temperature - default 0.2, and the author's advice is worth heeding: keep it low. LLaVA's accuracy degrades fast above ~0.2, unlike text-only Llama where people happily run 0.7.

The other two, model and mm_proj, are dropdowns populated from whatever GGUF files you've dropped in ComfyUI/models/llama/. mm_proj is the multimodal projection file that pairs with the model - they're separate files and they must match.

Installing it

The README's flow: clone into custom_nodes, then run its installer:

cd ComfyUI/custom_nodes
git clone https://github.com/ceruleandeep/ComfyUI-LLaVA-Captioner
cd ComfyUI-LLaVA-Captioner
python install.py

The install.py step is the part people skip and then complain about. It uninstalls and forcibly reinstalls llama-cpp-python with the right build flags - cuBLAS if it detects CUDA, Metal on Mac, plain CPU otherwise - because the pip-cached non-accelerated build would leave you on CPU inference. ComfyUI Manager will clone the repo but it can't run that custom build, so expect to run install.py manually either way.

Then the models. Download these into ComfyUI/models/llama/ from jartine/llava-v1.5-7B-GGUF:

• llava-v1.5-7b-Q4_K.gguf
• llava-v1.5-7b-mmproj-Q4_0.gguf

That's ~4-5 GB total. The model dropdown is empty until those files exist, so download first, then restart ComfyUI.

Common issues

• "LlavaCaptioner node not found" on a downloaded workflow - the pack just isn't installed. Install it via ComfyUI Manager's "Install Missing Nodes," or the git clone above. People hit this constantly; it's not a code bug.
• Inference times near 25s per image - you're on CPU. The README's own heuristic: GPU is 4-8s, CPU is 25s. If you're there, install.py didn't apply cuBLAS/Metal - check that the node actually built with GPU flags, then re-run the installer.
• Weird answers - bump max_tokens if captions truncate, and drop temperature toward 0 if you want accuracy. And don't expect NSFW replies from SFW images; the model's safety training plus image-conditioning makes that a dead end, by design.
• Swapped model variants - the README lists Obsidian, BakLLaVA, and ShareGPT4 GGUFs as alternatives, but each needs its own matching mm_proj. Mixing them gives garbage.