LLM Server Vision (GGUF)

A vision model is two files, not one: the model itself, and an mmproj - a small projector that turns image patches into something the language model can read. You already have the model loaded. This node serves the projector next to it, so the chat client can send a picture without a second model, a second port, or a second unload rule.

It's a config node and does nothing by itself. It emits a KINBURG_LLM_SERVER_VISION bundle you wire into Local LLM Server's vision input. One of three optional side models in the pack's llm_server package, each its own small node so the settings only exist when you use one.

Which door you actually want

Worth saying plainly, because it's the most common confusion with this pack. LLM Server Vision is for the chat client - SillyTavern, or anything else pointed at the gateway's base_url - so a person can paste an image into a conversation and get an answer about it. If what you want is a caption generated inside a graph (LoRA dataset captions, an img2img prompt seed, a judge scoring your render), that's the pack's Local LLM (GGUF) node with a Vision Settings (GGUF) config, not this. Same mmproj idea, different door.

And the honest ceiling from the captioning side of the community: vision-language models are decent describers and bad at keeping two people straight. They mix up who's wearing what as soon as a second subject is in frame. If you're captioning a dataset, that's the thing to audit by hand.

The inputs

mmproj is the projector .gguf, listed from ComfyUI/models/llm - the list walks subfolders and puts mmproj-named files first, which is a nice touch because a ComfyUI/models/llm/Qwen3-VL/ folder usually holds both the model and the projector. It must be the one built for this model. A projector from a different family loads and then produces fluent nonsense, which is worse than an error.

mmproj_path is the escape hatch when the dropdown shows (use model_path field).

mmproj_offload (default on) keeps the projector on the GPU. Turn it off - --no-mmproj-offload / --mmprojcpu on the backend - and the projector moves to the CPU: you get a little VRAM back and pay for it in slower image encoding. On a card that's also running a diffusion model, that's sometimes the right trade.

image_max_tokens (0 = the model's own limit) caps how much context a single image may eat. Lower is cheaper and blurrier; higher is a more detailed read of the picture at the cost of a chunk of your n_ctx. It's llama-server only. If your chat starts forgetting the beginning of the conversation once pictures enter it, this is the dial - a high-resolution image can take thousands of tokens, and they don't come back when the next message arrives.

Output: vision, wired into the server node's vision slot.

Install

ComfyUI Manager (search "Kinburg"), or:

cd ComfyUI/custom_nodes
git clone https://github.com/Kinburg/Kinburg-Nodes
# restart ComfyUI

Manager runs the pack's install.py, which installs the CUDA-matched llama-cpp-python wheel (0.3.30, chosen to match your torch's CUDA major - get that wrong and the import dies with the classic "llama.dll or one of its dependencies" message). The server binary is not bundled: grab llama-server or koboldcpp yourself and put it where the node's binary dropdown can find it, normally ComfyUI/models/llm. Model and mmproj .gguf files live in that same folder, which is why the recursive dropdown exists.

Nothing else. No second VRAM-hungry server, no extra Python.

Gotchas

The projector and the model must match, exactly. This is not a case where close is fine.

image_max_tokens is llama-server only, and backends that don't have a setting never get it silently - the server node lists what it ignored in its status output. Read that before concluding a widget did nothing.

Sending a picture isn't free just because the projector is tiny. The image turns into context tokens on every turn it's in the conversation, subject to image_max_tokens. One picture is fine; a picture per message in a long chat is how you blow through n_ctx.

An mmproj alone doesn't make a text-only model see. Some GGUFs simply have no vision counterpart; if the family never shipped a projector, there's nothing for this node to serve.