HunyuanVideo Vision Encode

HunyuanVideo 1.5's image-to-video isn't just "VAE-encode the first frame and call it a day." The model also runs your reference image through a vision encoder to get a set of semantic tokens - a compressed understanding of what's in the picture - and injects those into the transformer alongside the pixel condition. HyVideo15VisionEncode is the node that produces those tokens. It's the piece that lets 1.5 hold onto the subject's identity, composition, and style across the whole clip rather than only anchoring the first frame.

What it does

It takes the vision_encoder (from HyVideo15VisionEncoderLoader), resizes your reference_image to the model's target resolution (bucketed to supported sizes), runs it through the siglip encoder, and outputs vision_states (HYVID15VISIONSTATES) - a tensor of semantic tokens that feeds HyVideo15Transformer. If no reference image is provided (T2V), it fills the states with zeros, which is exactly what a text-to-video graph wants.

The settings:

• vision_encoder / hyvid_cfg / latents_dict - encoder, config (for task type and target resolution), and the latents (for batch shape).
• reference_image - optional IMAGE. Same image you're feeding HyVideo15VaeEncode and HyVideo15CFG.
• vision_num_semantic_tokens (729) and vision_states_dim (1152) - the token count and embedding width. These match the official recipe; change them only if you know what you're doing (e.g., matching a custom-trained projection).
• target_dtype (bfloat16) - matches the transformer's dtype.
• enable_offloading (true) - offloads the siglip encoder during its brief run. Keep on.

How it fits

Complete workflow: HyVideo15VisionEncoderLoader → HyVideo15VisionEncode → HyVideo15Transformer. In the simplified samplers this runs internally. It's also why 1.5's I2V is its strong leg - the semantic conditioning gives the model more than a pixel match to work from, which is part of why the distilled I2V clips look as cohesive as they do.

Installing it

Part of comfyui_hunyuanvideo_1.5_plugin. ComfyUI Manager → "HunyuanVideo-1.5 nodes", or:

cd ComfyUI/custom_nodes
git clone https://github.com/yuanyuan-spec/comfyui_hunyuanvideo_1.5_plugin
cd comfyui_hunyuanvideo_1.5_plugin
pip install -r requirements.txt

Restart. The vision encoder downloads via HyVideo15VisionEncoderLoader - and this is the piece of the pack that needs a HuggingFace token, because it pulls from the gated FLUX.1-Redux-dev repo.

Common issues

If this node returns error messages about the encoder or produces empty vision_states for an I2V run, check two things: that you actually passed a reference_image here (it's optional, so a missing wire doesn't error - it just silently switches the model into "T2V mode" and your I2V output ignores the image), and that the vision encoder downloaded properly (gated repo + no token = silent partial download). Also note vision_states carries the CFG duplication internally when guidance is on, so don't be surprised if the tensor is twice the batch size.