LLaVA (Managed Cache, Advanced)

Here's the workflow most people want: look at a reference image with a VLM to build a prompt, then immediately run a diffusion sampler with that prompt. The problem is that both models want the same VRAM, and ComfyUI doesn't always cooperate. This node is the answer - it's a full-featured LLaVa sampler that loads the model itself, and when it's done it can unload it, freeing your GPU for the actual generation.

What it is

LLavaOptionalMemoryFreeAdvanced is the pack's self-contained LLaVa node. Unlike the other samplers, it doesn't take a model from a separate loader - it reads the checkpoint straight from a dropdown and manages its own context, GPU offload, and thread settings. The "Optional Memory Free" in the name is the whole selling point: a boolean unload that, when checked, frees the model from memory after it answers.

The inputs that matter

• ckpt_name - dropdown of GGUF model files in models/LLavacheckpoints. The main model.
• clip_name - dropdown for the matching clip projector (mmproj-*). This pair must belong together.
• image - what the model looks at.
• system_msg and prompt - role and question, as usual.
• gpu_layers - how many layers to offload to the GPU (default 27). This is the setting that decides whether you're fast or VRAM-thin; if the model is crawling on CPU, raise it.
• max_ctx - context length, default 4096. Only raise it if you're feeding long prompts in.
• n_threads - CPU threads for the parts that don't run on GPU.
• temperature / top_p / top_k / penalties / seed - the usual sampling dials; defaults are sane.
• unload - the star. Default false. Check it and the model is released after generation, so your diffusion sampler gets the VRAM back.

Output is a single STRING.

How it works

Because the node owns model loading and unloads on demand, it's the pick for one-shot workflows: caption the image, grab the string, and let ComfyUI move on to the KSampler with a clean GPU. When unload is false it keeps the model cached, so repeated calls are fast but VRAM stays occupied - that's the trade you're choosing with one checkbox.

Install

Pack-level:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes

Restart (or Manager → "VLM Nodes"). Python 3.9+ and the llama-cpp-python stack. Model files go in models/LLavacheckpoints as a GGUF + matching mmproj pair from the same HF repo.

Troubleshooting

The llama-cpp-python wheel problems are the usual suspects - on Windows that's the "invalid wheel filename" install error and the missing llama.dll import failure people hit at startup; rebuild the package for your GPU backend and both clear up. If the model ignores your GPU entirely, that's the same root cause. If you're getting VRAM OOM at the sampler step, you forgot to tick unload. If captions are nonsense, check the GGUF/projector pairing before changing any dials.