ControlFoley Generate

This is the node you'll actually live in. ControlFoley Generate is the workhorse of the official ControlFoley pack: feed it a silent clip (or just a prompt) and it produces a 44kHz foley track that's supposed to match what's on screen. No API, no key, no cloud round-trip - a Xiaomi Research flow-matching model running locally inside ComfyUI.

Context helps here. Foley is the newest, thinnest layer of the ComfyUI stack - audio got bolted on only after video got good enough to want a soundtrack. The usual suspects (MMAudio, HunyuanVideo-Foley) do plain video-to-audio. ControlFoley's whole pitch is control: you can steer the sound with a text prompt, or pin its timbre to a reference audio clip. And control matters, because the hard part of foley was never making noise - it's making noise that lands on the right frames. Slightly mismatched foley breaks immersion instantly, which is exactly what the sync-conditioning here is aimed at.

What it does with what you connect

ControlFoley Generate is one node with three modes hiding behind which optional input you plug in:

• V2A: connect a video, leave prompt and reference audio empty. Pure video-to-audio.
• TV2A / TC-V2A: video plus a text prompt ("thunder strike", "skateboarding"). Same node path, both names in the docs.
• AC-V2A: video plus reference_audio_path - the generated sound mimics that clip's character. Keep the reference 2–4 seconds; longer gets truncated, shorter gets padded.
• T2A / TTA: no video at all, prompt only.

Mechanically, it's flow matching with a euler sampler running num_inference_steps (default fixed, which means 25). CLIP encodes your prompt, Synchformer plus visual features read the video, CAV-MAE provides sync embeddings, and in AC-V2A a CLAP/timbre path pulls the reference audio's character. All of that feeds a diffusion-style generator, and the waveform comes back through a BigVGAN vocoder as a native ComfyUI AUDIO tensor.

The inputs that matter

• prompt / negative_prompt - multiline. Empty prompt means the video alone drives the sound.
• num_inference_steps - a string: fixed (default 25) or any integer 1–100. Lower steps cut runtime, not peak VRAM, so don't treat them as a memory trick.
• guidance_scale - CFG strength, default 4.5, up to 20. This is the "how hard do you make it obey the prompt" dial.
• duration - an upper limit, not a promise. Text-only runs use 10s; video runs follow the input length, capped at 30s.
• clip_batch_size_multiplier and sync_batch_size_multiplier - frames per encoder call (batch size × multiplier), default 40. On a low-VRAM card, drop to 4–8 to shave peak memory.
• cache_video_features - default on; encoder output is cached so repeat runs of the same clip skip the heavy feature extraction.
• staged_offload - defaults to true, but here's the honest catch: the public upstream ControlFoley source doesn't implement it, so it's silently ignored with a console note. You don't need to fight it.
• reference_audio_path - the AC-V2A steering wheel, empty by default.

Optional inputs: video (from the ControlFoley Video Loader), video_input (native ComfyUI VIDEO), or images (an IMAGE batch, with image_fps setting the frame rate for a temporary MP4). Connect exactly one of the three.

Wiring it up

The node hands you four outputs: audio (the AUDIO tensor), sample_rate (INT), inference_time_sec, and peak_vram_gb (both FLOAT, handy for benchmarking). audio goes into ControlFoley Save Audio to write a WAV/FLAC, or into the Video-Audio Muxer to get an MP4 with the generated track replacing the original.

Gotchas

The whole pack is CUDA-only - no CPU, no MPS. First run downloads ~16GB of weights plus Hugging Face dependencies, so budget time. If huggingface.co hangs on your network, set HF_ENDPOINT to a mirror before starting ComfyUI. And note the low_vram flag lives on the Model Loader, not on this node - it's a text-only path, so if a video or reference audio is connected while it's on, generation throws a hard error rather than silently degrading. And install the pack through ComfyUI Manager ("ControlFoley Official") or git clone the repo into custom_nodes and pip install -r requirements.txt - just install only the missing packages one at a time, since that requirements file can quietly upgrade things your existing ComfyUI depends on.