CLIP Text Image Encode

Z-Image is a text-only model. Prompt goes in, image comes out, and if you wanted to say "no, in this style" while pointing at a reference, your options were a LoRA or trying to bolt an IP-Adapter onto something that was never built for it. This node is the different answer: it feeds the reference image straight into the text encoder itself, turning the encoder into a vision-language model so the picture becomes part of the prompt. No adapter, no extra cross-attention layers, no training. Just a much richer thing for the model to look at.

The name is doing a bit of a lie, in the grand ComfyUI tradition - there's no CLIP involved anywhere. The pack builds a real Qwen3-VL model (Qwen3-4B language model + the Qwen3-VL visual tower) and this node runs it. It's the inference-side version of the "teacher conditioning" trick from Tongyi-MAI's D-OPSD paper (on-policy self-distillation, May 2026): during training, a teacher VLM conditions the student on text plus image, and this node reproduces that exact path so your generation can be steered by a reference.

What you actually set

The inputs that matter, out of a schema with exactly two required and six optional:

• vl_model - the output of the pack's Load CLIP with Visual node. There's no working without it; it's the fused VLM this node runs.
• text - your prompt, and it's a proper multiline field with dynamic prompt support. Nothing CLIP-like here, no 77-token cliff; it's a Qwen chat turn.
• image1 / image2 / image3 - up to three reference images, each with an optional mask. Leave them empty and you get a plain text-only encode, which is handy for burning in a "no image" baseline.

Here's the mechanism, because it's genuinely interesting. The prompt and images get wrapped in the Qwen chat template as one user turn. The vision encoder runs first (the processor clamps each image to a 512×512 to 768×768 pixel budget), the image tokens are scattered into the text embedding sequence at the placeholder positions, and the language model does a full forward pass. The conditioning comes from the second-to-last hidden layer, padding tokens stripped, packaged as a normal ComfyUI conditioning. That's why the output wires into a KSampler exactly like the CLIP Text Encode output you're used to.

Two outputs:

• positive - the VLM embeddings of your text + images. Into the sampler's positive.
• negative - an empty-text, text-only encode. Into the sampler's negative.

One honest caveat before you get excited: if you're on Z-Image-Turbo at CFG 1 (the standard setup), the negative output does nothing, same as every negative on Turbo - it's guidance-distilled. On Z-Image Base at CFG 3–5, the negative actually has teeth.

The masks are worth a paragraph because they behave unexpectedly. A mask value of 0 zeroes those image-token embeddings before the LM reads them, and a token only survives if at least half of its patch is unmasked. So a tiny mask on a big image can silently vanish after downsampling. Use masks when you want a whole region ignored (a background you don't want bleeding in), not for fine edits.

Install and gotchas

Same pack as Load CLIP with Visual, so same steps - ComfyUI Manager (search "comfyui-zimage-vl") or:

cd ComfyUI/custom_nodes
git clone https://github.com/yaofeng/comfyui-zimage-vl

then restart ComfyUI. Two model files go in ComfyUI/models/text_encoders/ - the full story (including the visual-encoder extraction script) is on the Load CLIP with Visual page; the short version is you need qwen_3_4b.safetensors (which you already have if you run Z-Image) plus an extracted qwen_3_vl_visual.safetensors.

Where people get burned:

• Missing transformers. The repo ships no requirements.txt but imports transformers hard. If the node refuses to load, pip install transformers into the same Python that runs ComfyUI.
• VRAM. The VLM alone is ~9GB, on top of the 6B diffusion model. The README says 24GB, and that's not marketing - expect it to be the thing that makes a 12GB card fall over.
• Speed. First load is slow (it's building a 4B-parameter model from scratch), and the encode step is the bottleneck in every run after that. Batch your prompt exploration, don't iterate one image at a time.

Honest framing: this is a one-author, research-shaped pack with a tiny footprint - treat it as experimental, back up your workflows, and don't build a production pipeline on it yet. But as a way to actually point at a reference image with Z-Image, it's the most interesting option on the table.