Audio Gen Video 🎥AniPortrait

This is the reason anyone installs this pack: drop in a photo of a face and a voice recording, and AniPortrait_Audio2Video ("Audio Gen Video🎥AniPortrait") returns frames of that person talking, with mouth shapes and head motion roughly following the audio. It's the fully-local, no-API talking head - the kind of thing that went viral when AniPortrait shipped in early 2024, and the community's verdict back then was fair: it was the best open option for a while ("I've tried a few of these talking-head things and this is the best one yet"), even if it couldn't beat Alibaba's EMO on pure lip-sync polish.

How it works

AniPortrait is a two-stage pipeline, and this node runs both. First, the audio goes through a wav2vec2 encoder, then two small models translate it into motion: audio2mesh predicts a 3D face mesh (the lip shapes, the expression), and audio2pose predicts head pose. Those 3D outputs are projected down to 2D face landmarks. Second, a diffusion renderer - SD 1.5 with a temporal motion module, a reference UNet that injects your photo's identity, and a pose guider - draws your reference face performing that landmark sequence, frame by frame. It's the same motion-module-on-SD-1.5 lineage as AnimateDiff, focused on faces.

A couple of structural limits come straight from that design. Generation is chunked in 5-second pieces and the author caps inference at roughly 10 seconds of audio. And because the whole stack reloads from disk on every run, each generation has a long "loading models" phase before a single frame appears - expect patience, not interactivity.

Inputs that matter

• ref_image (IMAGE) - the photo to animate. Load it with Load Image; a clear, centered face is non-negotiable (it asserts "No face detected" otherwise).
• audio_path (Audio_Path) - from AniPortrait_Audio_Path. Give it this and the node runs the audio-driven path.
• height / width - 512×512 defaults, and square is the safe zone.
• seed / cfg / steps - seed 42, cfg 3.5, steps 25 by default. CFG behaves SD1.5-classically here, so don't crank it; low-to-mid is right.
• weight_dtype - fp16 (default) or fp32. fp16 is the sensible choice unless you're chasing NaN weirdness, in which case fp32 is the troubleshooting lever.
• accelerate (True) + fi_step (3) - the frame-interpolation speedup: it renders every Nth frame (film_net fills the gaps), so you can leave these on.
• length - 0 means "use the whole audio"; set a number to cap frames.
• images / fps - optional, and this is the hidden second mode: leave audio_path disconnected and feed a driving video's frames plus its fps here instead, and the node runs face reenactment (copying motion from a video rather than synthesizing it from audio). That's how the pack's face_reenacment_workflow.json uses it.

The long list of *_path enums (vae_path, model, motion_module_path, denoising_unet_path, etc.) are pre-filled from the pack's config - you should never have to touch them.

Output and finishing

It outputs images (IMAGE) - and note it is not an output node, so nothing is saved until you wire it into VHS Video Combine. Do that, and feed the combine's audio input from AniPortrait_Audio_Path's VHS_AUDIO output to get sound back on the clip.

Install - the heavy one

The pack install is easy; the model downloads are the actual project:

cd ComfyUI/custom_nodes
git clone https://github.com/frankchieng/ComfyUI_Aniportrait
cd ComfyUI_Aniportrait && pip install -r requirements.txt

Then create a pretrained_model/ folder inside the pack directory and drop in: Stable Diffusion 1.5 (runwayml/stable-diffusion-v1-5), sd-vae-ft-mse, the CLIP image_encoder from lambdalabs/sd-image-variations-diffusers, wav2vec2-base-960h, plus the AniPortrait weights from ZJYang/AniPortrait on HuggingFace - denoising_unet.pth, reference_unet.pth, pose_guider.pth, motion_module.pth, audio2mesh.pt, audio2pose.pt, and film_net_fp16.pt. Each subfolder/weight goes in the exact path the config expects, which is where people get burned - the pack's docs are famously thin about folder structure.

Common problems

• "No face detected." Your reference image. Face must be clearly present, and square-ish framing helps.
• Weird output on landscape clips. Square input, always.
• Audio longer than ~10s just doesn't render past the cap - cut the clip or use seek_seconds on the Audio Path node to trim.
• Wrong diffusers. The pack pins diffusers==0.26.2; older versions crash on a class rename (PositionNet → GLIGENTextBoundingboxProjection). If you're on an old env, pin the version in requirements.txt.

If you just want a quick talking head in 2026, the WanAnimate / LTX-2 world is objectively better. But if you're here, you're probably on an SD 1.5 box or re-running a classic workflow - and this node still does what it says.