Vision-SLM: Generator

SLMVisionGenerator ("Vision-SLM: Generator") is the node this pack is named for: it takes an image straight from your ComfyUI graph, a prompt, and a vision-language model object, and returns the model's answer as text. Feed it the output of a KSampler or an image loader, ask "describe this in detail for a prompt," and the caption comes back as a STRING you can feed anywhere - into a text encoder, into a save node, into another prompt stage.

This is the local, no-API version of what a lot of people run through cloud vision models. The models here are small (DeepSeek-VL 1.3b through Qwen3-VL-4B and Gemma-3-4b), so expectations should match: they're great for captioning, object counting, and quick content questions on your own renders. They are not frontier-grade reasoning, and they won't run fast on a potato.

How it works

The generator's job is mostly plumbing, and it's done carefully. First it converts the ComfyUI IMAGE tensor into a PIL image - if your batch has multiple frames it silently uses the first one, so don't feed a video batch expecting per-frame analysis. The model is loaded (or reused from the single-resident cache) and generates with your sampling settings; the pack's backends handle each model family's chat template and, for the Qwen3-Thinking and other reasoning models, strip the <think> block so you get the answer, not the internal monologue.

The inputs that matter

• image - the IMAGE tensor. Required.
• slm_model - the object from "Vision-SLM: Model Selector."
• user_prompt - what you're asking about the image. Wire this from the pack's "Prompt+RAG" node to give the model context.
• max_new_tokens (1024), temperature (0.85), top_p (0.9), top_k (0), repetition_penalty (0.0) - standard sampling knobs; zero means "not applied."
• use_model_cache - default on; one model resident in VRAM at a time.

One output: SLM_TEXT, a STRING.

Install

In the comfyui_pilcothink_VisionSLM pack:

cd ComfyUI/custom_nodes
git clone https://github.com/gpdev-Pilcothink/comfyui_pilcothink_VisionSLM

Restart, or install via ComfyUI Manager (search comfyui_pilcothink_VisionSLM). Heavy requirements - transformers ≥ 4.56.2, torch, accelerate - and the first run downloads whatever vision model you pick into the pack's own Models/VisionSLM/ folder.

Gotchas

First run is the painful one: multi-gigabyte downloads on first select, and if you pick gemma-3-4b-it it's gated on Hugging Face, so the auto-download 401s until you sort out a token or place the weights manually. Batch images: only the first frame is analyzed - that's a feature of this node, not a bug, but it surprises people. And remember the pack's model-cache policy: only one model stays loaded, so switching models between runs means an unload/reload churn that eats time. If use_model_cache is off, it unloads after every single run, which is brutal for iteration speed - keep it on unless you're juggling VRAM.

One honest note on the ecosystem: LLM-vision custom nodes are exactly the category that produced the ecosystem's biggest malware incident, and this pack is brand new with no track record. Install from the official repo, and maybe give it a spin in a workflow you don't care about before trusting it with your main setup.