🖼️ V2 Local Vision Analysis

This is the node that makes the "local GGUF mode" from the pack README real: V2 Local Vision Analysis takes a VISION_MODEL from the loader, an image or video-frame batch, and a prompt, and runs the whole inference in-process via llama-cpp-python. No server, no port, no network. If you want a vision model that lives entirely inside ComfyUI, this is the runner end of the deal.

It's the local counterpart to V2 Remote Vision Analysis - same "ask the model anything" philosophy, but with the model loaded into your own process instead of sitting behind an OpenAI-compatible endpoint. Which one you want comes down to a trade: local means zero server management and no separate app, but it pins VRAM and puts llama-cpp-python's build (CUDA or not) in charge of your speed. Remote lets the model live in LM Studio with GUI management. For a 3B model on a modest card, local is fine; for an 8B, think about which side of that trade you prefer.

The inputs that matter

• model - required VISION_MODEL from V2 Local Vision Model Loader.
• prompt - your instruction. Default "Describe this image in detail."
• max_tokens - default 512 (note: lower than the remote node's 1024). Bump it for detailed captioning.
• temperature / top_p / top_k / seed - the full sampling stack. Defaults are 0.7 / 0.9 / 40. For reproducible analysis you want the seed set and temperature low; these knobs exist because llama.cpp gives you the whole sampler and the pack exposes it.
• image or video - either works, and the tooltips are explicit that it's one or the other. Both are IMAGE type (video = frame batch). This node accepts a video frame sequence directly, which is the local path to video understanding.
• system_prompt - optional, defaults to an "accurate and detailed" assistant persona.

How it works

The loader hands this node an already-loaded llama-cpp-python model plus its mmproj, the node converts your image or frames into the right input format, runs inference with your sampling parameters, and returns a context STRING. It's an output node, so the description renders in the UI. The heavy lifting is done at load time - if loading took a while, that was VRAM and model init, not this node.

The honest catch

Same one as every local GGUF VLM: the mmproj. If the model "can't see" your image and describes a black void or refuses, it's almost always the projector file, not your prompt. The loader auto-finds and auto-downloads it, but if you sourced the GGUF manually, verify the mmproj matches the model family - this is the most common "why is my local vision model blind" post in the llama.cpp community. Also, quant level matters for detail work: Q4 reads big text fine, but small or stylized text in images is where Q8 earns its extra VRAM.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/walke2019/ComfyUI-GGUF-VLM.git
cd ComfyUI-GGUF-VLM
pip install -r requirements.txt

Restart ComfyUI; the node is under GGUF-VLM → v2 → Local. The dependency that actually runs it is llama-cpp-python, so make sure it installed with the acceleration you expect (CUDA on Windows, or a CUDA rebuild on Linux if you want GPU speed).

Troubleshooting

Gibberish output usually means the wrong mmproj or a bad quant for the task. Slow output on a GPU machine means llama-cpp-python is running CPU-only - check your install. A context overflow when you feed a long video batch is the frame count fighting n_ctx from the loader; sample fewer frames upstream (Video Frame Sampler) or raise n_ctx. And if you loaded a ✗ model, remember it downloaded at click-time, so the first run can look "frozen" while it pulls gigabytes.