LLaVA Sampler

If you want to know what's in an image and you don't want to fight a wall of sliders to find out, this is the node. LLavaSamplerSimple is the pack's stripped-down LLaVa captioner: wire in an image, type what you want to know, pick a model, hit queue. Four inputs, one output, done.

What it is

It's the simple variant of the pack's LLaVa samplers, in the same family as the more configurable Advanced version. It runs a LLaVa vision-language model - the GGUF flavor - over your image and returns the model's answer as text. Classic uses: captioning an input image to build a prompt, describing a reference so you can feed the description into a prompt generator, or just asking questions about a picture. The README notes the whole prompt-generation side of the pack works best with LLaVa 1.5 and 1.6, so those are the models to reach for.

The inputs that matter

• image - what the model looks at.
• prompt - your instruction. "Describe this image in detail," "what style is this?", whatever. It's a STRING, so plain text.
• model - a CUSTOM input: the GGUF LLaVa checkpoint you loaded through the pack's loader node. This is the "simple" trade - the sampler itself doesn't load models, a loader does.
• temperature - default 0.1. Lower = more literal and consistent; higher = looser, more creative descriptions.

Output is a single STRING. Send it to a text display, a SimpleText node, or straight into JsonToText if your model answers in JSON-looking text.

How it works

The sampler wraps llama.cpp inference over the model: your image gets embedded, your prompt gets sent alongside a system message ("You are an assistant who accurately describes images"), and the model generates text token by token. Because it's llama.cpp, the heavy lifting - GPU offload, context window - is handled by the loader's settings, not by this node. Keep it simple, keep it honest.

Install

This one carries the pack's real install burden, so read carefully:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes

Restart ComfyUI (or use Manager → search "VLM Nodes"). You need Python 3.9+, and the llama.cpp stack (llama-cpp-python). Then drop two files into models/LLavacheckpoints:

• the model GGUF - e.g. ggml-model-q4_k.gguf
• its matching clip projector - mmproj-model-f16.gguf

Every model's clip projector is different. Grab both from the same HF repo (LLaVa 1.6 Mistral 7B, Nous Hermes 2 Vision, LLaVa 1.5, BakLLaVa are all listed in the README) or the image understanding will be garbage in a way that's hard to diagnose.

Troubleshooting

The classic failure modes are llama-cpp-python, not this node. On Windows, people hit "invalid wheel filename" during install or a missing llama.dll import error on startup - reinstall llama-cpp-python with the correct backend build (CUDA/Metal/ROCm) from the abetlen docs and those go away. If your model loads but runs on CPU, that's the same cause: llama-cpp-python wasn't built for your GPU. And if output is nonsense, check that your GGUF and mmproj actually belong together before you touch any other setting.