ClipProj Generate / Caption

Here's the thing people miss about the ComfyUI-ClipProj approach: the Qwen3-VL you load as a text encoder is, underneath, a full vision-language model. It can generate text - it just usually never gets the chance. This node hands it back the job. Same resident weights, no second model loaded, one extra node in the graph.

The pack ships two example workflows that show why you'd want it. One rewrites a one-line idea ("an old fisherman mends his net on a quay at sunrise, 10 seconds") into a full three-shot MiniMax H3 prompt, with subject definitions and timecoded shots. The other captions an image - on the same encoder that's doing your conditioning. It's the "LLM as a tool in the graph" pattern the ecosystem has converged on, except the tool was already sitting in VRAM.

Why this node had to exist

ComfyUI's built-in SDClipModel.generate quietly breaks captioning on Qwen3-VL: it drops embeds_info and never calls build_image_inputs, so image tokens end up at linear positions instead of Qwen3-VL's 3D mRoPE, without the DeepStack injection. The result is a caption that reads like the model never saw the image. This node restores the full path - which is why the author warns people off "just use Generate" on a stock load.

The inputs

• clip - the same encoder you're conditioning with. Wire it from the ClipProj loader (or the stock Load CLIP + Apply chain).
• system / prompt - the chat template, defaulting to a helpful assistant and "Describe this image." Multiline, so you can paste a whole H3 system prompt in.
• max_length - output cap, default 256, up to 4096.
• temperature (0.7), top_p (0.9), top_k (50), seed - the standard sampling knobs.
• Optional image - the thing to caption. This is the input that makes the node a captioner rather than a bare text generator.
• Optional precision - weights (default) follows the loaded model's dtype; the others force float16/bfloat16/float32. ComfyUI runs float16 weights in bfloat16 and re-casts everything on every token, which is a big chunk of why generation is slow here.
• Optional preload_head - prepares the output matrix once instead of re-casting it per token. Leave it on unless VRAM is tight.
• Optional repetition_penalty (1.05) - discourages loops, with a real caveat, below.

Output is text (STRING) - wire it into a text display node, or straight into a prompt slot downstream.

Set expectations on speed

This is a convenience, not a replacement for an inference server. Generation runs far slower than llama.cpp or vLLM - ComfyUI's ops re-cast weights on every forward, which is fine for diffusion and painful when decoding one token at a time. The author quotes around 20 tok/s on a 3090, with precision and preload_head as the two mitigations exposed. For heavy or batch use, spin up a real engine and feed the output back in. For the occasional "turn this sentence into a proper H3 prompt," it's more than enough.

The repetition-penalty trap

repetition_penalty only applies while sampling. At temperature 0, ComfyUI takes the most likely token outright and ignores it - so the parameter you set to cure a loop does nothing precisely when you need it. The author's recipe: raise temperature to about 0.3 first, then push the penalty toward 1.2. If you're locked in a repeating phrase, that's the order to do it in.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/nicolab28/ComfyUI-ClipProj

Restart ComfyUI (Manager: search "ComfyUI-ClipProj"). No requirements.txt, no pip install - torch and ComfyUI's own modules only. Needs ComfyUI 0.31.0+. The node carries no weights of its own; it reuses whatever encoder you point it at, so there's nothing extra to download to make it work.