FOMM Runner

If you've ever wanted to take a still photo of a face and make it do things - talk, nod, shift expression - this is the node the whole pack is named after. FOMM Runner is a ComfyUI-native port of the First Order Motion Model (Siarohin et al., NeurIPS 2019), the paper that basically defined "image animation" before anyone talked about video diffusion. You feed it one photo plus a driving video, and it transfers the video's motion onto your photo. The output is 256×256 and it takes seconds, not minutes. It's a GAN, not a diffusion model - there's no sampler, no CFG, no prompt, and nothing to tune until it looks "right." You just run it.

That's both the charm and the limit. Modern alternatives like LivePortrait or SCAIL-2 run rings around it on fidelity and resolution. But those need big weights and a real GPU budget; FOMM is a ~200MB checkpoint that runs on hardware a diffusion model would laugh at. For quick talking-head tests, style transfer onto a face, or as a frame source before upscaling, it's genuinely still useful.

How it works

The mechanism is simple enough to describe in one breath. A keypoint detector finds ten keypoints on the source face, each with an associated local Jacobian. It does the same for the first frame of the driving video. Then, frame by frame, it takes the difference between the driving frame's keypoints and the driving start frame's keypoints, adds that displacement to the source keypoints, and lets a dense-motion network warp the source image to match. An occlusion-aware generator fills in whatever gets revealed. That's the whole pipeline - the code is right there in inference_fomm.py and it's ~100 lines.

The inputs that matter

Most of the toggles come straight from the original repo, and the author's tooltips are honest about them:

• model_name - vox-adv (default) or vox. vox is the paper's original checkpoint; vox-adv is the "advanced" retrain and generally looks better. Stick with the default.
• relative_movement - on, you inherit the proportions of the driving face; off, the source face gets bent to match the driver's geometry exactly. Leave it on.
• relative_jacobian - only does anything when relative_movement is on, and the README says leaving it on is what stops the face deforming "in a freaky way." On.
• adapt_movement_scale - on, it scales the motion by the relative size of the two faces. Turn it off and the source face gets heavily distorted to match the driver's movement. On.
• find_best_frame - searches the driving video for the frame whose keypoints best match the source, then generates outward from it. "Gives mixed results," per the author, and it only works if you've installed the optional face-alignment library. Without it the node silently uses frame 0.

source_image is a single still; driving_video_input is a batch of frames, not a video file - you get that from VHS_LoadVideo (the included workflows use VideoHelperSuite). The optional audio socket is pure pass-through: the node never touches it, it just hands it back to you so you can wire it into VHS_VideoCombine along with images and the frame_rate output.

Install

The whole pack installs at once, and ComfyUI Manager can do it - search "FirstOrderMM" - or by hand:

cd ComfyUI/custom_nodes
git clone https://github.com/FuouM/ComfyUI-FirstOrderMM.git
pip install -r requirements.txt

Dependencies are refreshingly light: numpy, torch, scipy, pyyaml, matplotlib, einops, timm. No OpenCV, no PyAV. Restart ComfyUI and the nodes appear under the FirstOrderMM category.

Here's the trap: nothing downloads automatically. The checkpoints must go in the pack's checkpoints/ folder, and the filename matters - this node loads vox-adv-cpk.pth.tar and vox-cpk.pth.tar. Grab them from the original first-order-model repo or graphemecluster's first-order-model-demo release. No checkpoints, no output, and the error you get is a bare file-not-found, not a friendly hint.

Common issues

• "File not found" on load - you haven't placed the checkpoint, or it's named wrong. checkpoints/vox-adv-cpk.pth.tar, exactly.
• Wonky, rubbery faces - one of the three movement toggles got flipped. All three on (the defaults) is the sane starting point.
• find_best_frame does nothing - you need pip install face-alignment (plus its model downloads) for that toggle to do anything at all.
• Blurry output - 256×256 is the internal resolution, always. Plan to upscale after, or accept it as a preview.