HalloNode

You've got a portrait and a voice memo, and you want the person in the picture to actually say it. That's the entire job of HalloNode: feed it one face image and one audio clip and it hands back an mp4 where the mouth, the head pose, and the expressions follow the audio. It's the ComfyUI port of Hallo, the audio-driven portrait animation model from Fudan University, and it runs entirely on your machine - no API, no key, nothing leaves your box. The catch is the price of admission, which we'll get to.

How it actually works

Hallo is a video-diffusion pipeline bolted onto Stable Diffusion 1.5. Under the hood there's an AnimateDiff-style motion module (mm_sd_v15_v2.ckpt) for temporal coherence, a wav2vec2 audio encoder that turns the track into embeddings taken from 12 hidden layers, a face-analysis stage that locates the face and builds the region to animate, and even a vocal separator (Kim_Vocal_2) that isolates speech from backing music so the lip-sync isn't thrown off. Sampling is DDIM with a zero-SNR v-prediction schedule, 40 steps by default, output encoded as h264 at 25 fps.

One mechanism detail worth knowing: when you hit Generate, the node writes a temp YAML config and shells out to Hallo's own inference script in a separate process. That's why the ComfyUI UI locks up while it grinds, and why a crashed run can leave a stray python process chewing your GPU. It also means this is not prompt-driven - no text influences anything. The face, the audio, and three weight knobs are all you get.

The inputs that matter

Only two inputs should ever get your attention on first run:

• source_image - a single face photo. Front-facing, decent light, eyes open. Hallo is pickier than your average checkpoint.
• driving_audio - the clip you want spoken, any length you're willing to render.

The three weights - pose_weight, face_weight, lip_weight - control how strongly motion follows the audio. All default to 1, and you can genuinely leave them alone until something looks off. face_expand_ratio (default 1.2) decides how much of the region around the face gets animated; drop it toward 1.0 if you see the jaw and neck warping.

The output is a single VIDEO, which you preview with PreViewVideo - the pack's own output node - or with any of the community video-preview nodes.

Install and the model-download gotcha

Install is standard: ComfyUI Manager, search ComfyUI-Hallo, install. Or the manual route:

cd ComfyUI/custom_nodes
git clone https://github.com/AIFSH/ComfyUI-Hallo.git
cd ComfyUI-Hallo
pip install -r requirements.txt

You also need ffmpeg on your PATH (apt install ffmpeg on Linux, WingetUI on Windows) and an xformers build matched to your torch version - the README calls out xformers==0.0.22.post7 for torch 2.1.0+cu121. Then the sneaky part: on first import the pack auto-downloads the entire fudan-generative-ai/hallo snapshot from Hugging Face into pretrained_models/ - SD1.5 base, motion module, wav2vec2, VAE, face analysis, several GB all told. It happens at load time, so the first startup can look like ComfyUI hung. Let it finish; the console shows progress.

Where people get burned

• "No module named 'hallo'" after install. The pack injects a .pth file into site-packages so its internals import - if you hit this, just restart ComfyUI.
• Red "IMPORT FAILED" on the node. Almost always a missing pip dependency. Read the console - it names the actual module - and pip install it. The dependency list here is heavy (onnxruntime, insightface, mediapipe, bitsandbytes, decord…), and this pack is a known source of ComfyUI dependency hell precisely because it's so big.
• VRAM. Figure 8 GB as the realistic floor. People have run it on 6 GB, but they weren't happy about it.
• Speed. On a 4090, expect roughly a minute of render per second of video. That's not hyperbole - it's the model's reputation. This is a "start it, go make coffee, come back" node, not a realtime toy.

HalloNode is the best local option in its niche, but it earns its keep slowly. If you need fast turnaround on talking heads, cloud tools are the practical alternative; if you want it local and patient, this is your node.