AniPortraitLoader

AniPortraitLoader is where AniPortrait happens before you ever see a frame. It's the node that takes nine file paths, loads an entire diffusion stack into VRAM, and hands you a custom pipe object plus an a2m_model that only one other node in the world knows how to use: AniPortraitRun.

AniPortrait itself is Zejun-Yang's audio-driven portrait animation project (paper 2403.17694, released March 2024). The pitch: give it a reference portrait, an audio clip, and a driving video of someone moving their head, and it makes the portrait talk with the audio and mimic the head motion. This ComfyUI port by chaojie (who also ports DynamiCrafter and MuseV) splits that into a loader node and a run node - and this loader is the heavy half.

How it works

Read the loader's source and you'll see it isn't one model, it's six or seven strapped together. The pipe it returns is a Pose2VideoPipeline built from:

• A VAE (sd-vae-ft-mse) and a 2D reference UNet - the SD 1.5 UNet that holds the identity of your portrait, same idea family as the reference-net/IP-Adapter tricks covered elsewhere in the KB.
• A 3D denoising UNet - the SD 1.5 UNet with an AnimateDiff-style temporal motion module bolted on (the inference config declares a "Vanilla" motion module with temporal self-attention). That's the same lineage as AnimateDiff: freeze the image model, add temporal attention, get video.
• A pose guider - a ControlNet-like conditioner that injects the landmark skeleton into the denoising UNet so the face follows the pose frames.
• A CLIP vision image encoder (from sd-image-variations-diffusers) that encodes the reference photo.
• A DDIM scheduler configured with v-prediction and zero-SNR settings - the config even flips prediction_type to v_prediction, which is a deliberate choice, not a bug.

The a2m_model output is the audio side: a wav2vec2 encoder plus the trained audio2mesh.pt weights that turn speech features into a 3D head mesh. Both outputs get loaded onto CUDA and left there.

The inputs that matter

Honestly, it's nine path strings. Set them once and forget them. The defaults are the author's own machine (/home/admin/ComfyUI/...), so every one of them is wrong on your box until you fix it.

• sd_path, vae_path, image_encoder_path, wav2vec2_path - these are diffusers-format model directories, not single .safetensors files. stable-diffusion-v1-5, sd-vae-ft-mse, image_encoder (from sd-image-variations-diffusers), and wav2vec2-base-960h all live as folders under ComfyUI/models/diffusers/.
• a2m_ckpt, motion_module_path, denoising_unet_path, reference_unet_path, pose_guider_path - the five trained AniPortrait weights from HuggingFace (ZJYang/AniPortrait): audio2mesh.pt, motion_module.pth, denoising_unet.pth, reference_unet.pth, pose_guider.pth.
• weight_dtype - fp16 by default, and it's the right call on most cards. Flip to fp32 only if fp16 gives you NaNs or you've got VRAM to burn.

The outputs are pipe and a2m_model. Both go straight into AniPortraitRun's matching sockets.

Installing it

The pack installs the usual ComfyUI way - Manager, search "ComfyUI-AniPortrait", or:

cd ComfyUI/custom_nodes
git clone https://github.com/chaojie/ComfyUI-AniPortrait

Then the part people skip at their peril: pip install -r requirements.txt. That list is brutal - decord, av, mediapipe, opencv-contrib-python, onnxruntime-gpu, librosa, controlnet-aux, diffusers>=0.26.3, and a clip package pip-installed straight from a GitHub archive. The code also hardcodes custom_nodes/ComfyUI-AniPortrait as the path to its own configs, so don't rename the folder after cloning. Everything runs on GPU; the README suggests CUDA 11.7 and Python ≥3.10. And note that because the whole nodes.py module imports at load time, none of the pack's nodes - not even the lightweight helpers - will appear until that requirements file actually installs.

Where people get burned

The number-one community complaint back when this shipped (the pack's own release thread on r/StableDiffusion, April 2024) was that nobody knew what to do with wav2vec2-base-960h - because it's a whole folder, config.json and pytorch_model.bin and all, not a single weight file. Same for image_encoder. If your loader throws and you're pointing at one file, that's why.

Two more things worth knowing. There's no model caching: every run of this node rebuilds and re-loads the whole stack, so changing a path costs you a long reload, and just re-running the workflow can eat minutes before diffusion even starts. And it's a hungry beast - SD 1.5, a CLIP vision encoder, two UNets, a motion module, and a mediapipe model in fp16 is a lot of a 12 GB card. If ComfyUI crawls after this node, that's the price of admission for the 2024-era talking-head pipeline. It still works fine - it's just not the newest game in town anymore.