HunyuanFoley Visual Encode

HunyuanFoley Visual Encode is the node that makes this a foley pipeline instead of just another text-to-audio generator. Foley is sound that syncs to picture, and this is where your picture gets in. It takes a batch of video frames, reads the fps you tell it, and produces two feature sets plus a duration - the exact conditioning the Denoiser needs to generate audio that tracks the on-screen action.

It's part of Vantage-HunyuanFoley, a modular rework of Tencent's HunyuanVideo-Foley that splits encoding and sampling into separate nodes so you don't hold the whole stack in VRAM at once. The original monolithic workflow kept SigLIP2, Synchformer, CLAP and the diffusion model resident together, which is rough on 8–12GB cards. This node does its encoding work, hands you CPU tensors, and frees the GPU for the next stage.

How it works

Two models run inside, each with a different job:

• SigLIP2 (google/siglip2-base-patch16-512) extracts the content features - what's in the frame, semantically - sampled at 8 fps, returning the model's pooled embeddings.
• Synchformer (synchformer_state_dict.pth, the audio-visual synchronization model) produces the timing embeddings that keep the audio aligned with the video, sampled at 25 fps.

The node resizes frames to 224×224, samples them on even intervals based on your fps, runs both encoders under no_grad, then offloads and hands you the features on CPU. The third output, audio_len, is just frame_count / fps in seconds - it's computed, not something you set, and it's how the Denoiser knows how long a clip to generate.

The inputs you care about:

• video_frames (IMAGE) - standard ComfyUI frames, normalized 0–1. In the example workflow these come from VHS_LoadVideo, which is also where the pack's muxing gets its frames.
• fps (float, default 24) - plug VHS_VideoInfo.loaded_fps in here rather than typing it, because it silently drives the audio length downstream.
• seed (int) - for reproducible feature-extraction order and any RNG use.
• to_gpu / move_back_to_cpu - the usual offload toggles.

Outputs are siglip2 (SIGLIP2_FEAT), syncformer (SYNC_FEAT) and audio_len (AUDIO_LEN), all feeding the matching inputs on the Denoiser.

Installing it

Same pack as its siblings: ComfyUI Manager → search Vantage-HunyuanFoley → install → restart, or clone manually and pip install -r requirements.txt. Then download assets into ComfyUI/models/hunyuan_foley/:

mkdir -p ComfyUI/models/hunyuan_foley/siglip2
# synchformer_state_dict.pth  ->  models/hunyuan_foley/
# siglip2/model.safetensors, config.json, preprocessor_config.json

Both the siglip2/ folder (from google/siglip2-base-patch16-512) and the Synchformer state dict must exist exactly where the node looks, or it raises a clear "folder not found" / "state dict not found" error.

Where people get burned

The classic failure is the same one that plagues other foley nodes: the encoding "works" but the result is audio that has nothing to do with your video, because the frames never actually reached this node or the fps was wrong. fps matters twice - it sets the sampling cadence and the generated clip's length - so wire it from VHS_VideoInfo and let the graph stay honest. On a 3060-class 12GB card this node is fine with move_back_to_cpu on; the big VRAM spike is the Denoiser, not this, so give it the offload toggle and move on.