ControlFoley Model Loader

Every serious ControlFoley workflow starts here. ControlFoley Model Loader loads the actual model - the ControlFoley generator plus its feature encoders - from your local source and weight directories, caches it in memory, and hands you a controlfoley_model object that every generation node and the muxer consume. It's the one node that's always in the chain, and it's where your GPU budget gets decided.

What loads and why it's heavy

ControlFoley is a flow-matching video-to-audio model, but this node doesn't just load one network. It builds the runtime with the CLIP text encoder, the Synchformer and visual-feature extractors, the CAV-MAE sync encoder, and the CLAP/timbre path for reference audio, plus the generator and the BigVGAN vocoder. That's why the five weight files total ~16GB (about 11GB core + 5GB external encoders). First load therefore takes minutes even on a fast pipe - the README is blunt that later runs are dominated by model construction and GPU transfer, not downloads.

Missing weights are fetched from Hugging Face on execution. If huggingface.co hangs on your network, set HF_ENDPOINT to a mirror before starting ComfyUI - otherwise the first load can look like it's wedged.

Inputs that matter

• controlfoley_source_dir / model_weights_dir - where the source tree and weights live. auto_fetch_source (default on) clones the pinned upstream revision into <ComfyUI root>/controlfoley if it's missing. Leave model_weights_dir as path/to/model_weights to use the default auto-download directory.
• variant - only large_44k exists right now (the 44kHz model), so this is decorative until more variants ship.
• device - auto or cuda. The public inference path is CUDA-only; CPU/MPS just won't run.
• precision - bf16 (default), fp16, fp32. bf16 is the sensible middle for consumer cards; fp32 if you want reference quality and have the VRAM.
• low_vram - the text-only memory path. Keep it false for V2A, TV2A, TC-V2A, or AC-V2A; the generation node hard-errors if you combine low-VRAM with video or reference audio.
• compile_encoders - default false. Turning it on pays a one-time torch.compile cost for faster encoder passes. Leave it off for your first smoke test.

The optional dependencies input accepts the controlfoley_dependencies object from Load ControlFoley Dependencies - plug it in and the loader adopts the already-resolved source/weight dirs and low-VRAM setting instead of re-resolving them.

Output

One output, controlfoley_model, of type CONTROLFOLEY_MODEL. That single wire feeds ControlFoley Generate, ControlFoley Advanced Generate, Torch Compile, the Video-Audio Muxer, and the Unloader - and the loader's cache is the whole reason the advanced chain beats Simple Generate on repeat runs. It's also the one to blame if you get "This ControlFoley model was unloaded" from a generate node: re-run the loader (or restart ComfyUI).

A small design nicety worth knowing: the loader's IS_CHANGED forces a reload after any unload, so the Unloader can't leave a stale gutted runtime in the graph. Install the pack through ComfyUI Manager ("ControlFoley Official") or git clone YJX-Research/comfyui-controlfoley-official into custom_nodes + pip install -r requirements.txt, installing only genuinely-missing packages one at a time - that requirements file carries version ranges that can upgrade shared deps like transformers or numpy under you.