FSRT Runner

FSRT Runner is the odd one out in this pack, and it's the one you want when quality actually matters. FSRT - Facial Scene Representation Transformer (Rochow et al., CVPR 2024) - is a transformer-based face reenactor, the newest architecture here and the only one that doesn't think in 2019 terms. Everything else in ComfyUI-FirstOrderMM is a fast GAN that warps pixels around. FSRT actually re-renders the face: it separates appearance, head pose, and expression into distinct features, then draws the output pixel by pixel. That's why it looks better, and also why the README's warning is blunt: "This model takes the longest to run. The full Damedane example takes ~6 minutes." On the same input where FOMM finishes in seconds.

So use it when the GANs aren't cutting it - more natural motion transfer, better handling of pose differences, cleaner identity. It's still not diffusion-quality, but within this pack it's the ceiling.

How it works

A keypoint detector extracts keypoints from source and driving frames, and an expression encoder pulls out the expression features. The encoder then builds a compact "scene representation" of the source, and the decoder renders the reenacted face - chunk by chunk, since it's processing pixels rather than warping a whole image at once. The max_num_pixels knob is literally how many pixels it processes in parallel per chunk, which is why it's both the VRAM lever and the speed lever.

The inputs that matter

• model_name - vox256 (default) or vox256_2Source. The 2Source variant is trained on vox256 + celebhq, and per the README it's the one that handles source/driver identity mismatch better. If you're animating a face that isn't in the training set, try it.
• relative - relative keypoint coordinates (scale motion to the source) versus absolute (map straight onto the driver). Default is false, unlike FOMM - flip it if you're getting distortion from mismatched face sizes.
• adapt_scale - scales movement by the convex hull of the keypoints, i.e. adapts motion to the relative size of the faces. Default false.
• find_best_frame - same feature as FOMM, same requirement: needs pip install face-alignment or it silently starts at frame 0.
• max_num_pixels - default 65536. This is your OOM escape hatch. Run out of GPU memory? Halve it. The render loop just processes more chunks. It's the only "performance" setting here that actually changes memory usage.

Outputs are the pack standard - images, audio (pure pass-through), frame_rate - and they wire straight into VHS_VideoCombine. Note that FSRT returns predictions without the shared out_video() helper; the node concatenates the tensors itself, but from the graph's point of view it behaves identically.

Install

Same pack, same install - ComfyUI Manager (search "FirstOrderMM") or:

cd ComfyUI/custom_nodes
git clone https://github.com/FuouM/ComfyUI-FirstOrderMM.git
pip install -r requirements.txt

Light deps (numpy, torch, scipy, pyyaml, matplotlib, einops, timm), restart, nodes under FirstOrderMM.

The model situation is where FSRT differs, and it's the one real trap. You must download kp_detector.pt as well as the model. The dropdown only lists vox256 and vox256_2Source, but the keypoint detector is a separate file and the README is explicit: "For FSRT, you must download kp_detector." All three go in module_fsrt/models/. Forget the kp detector and the node fails at inference with an obscure load error.

Common issues

• OOM at load or mid-video - drop max_num_pixels from 65536 to 32768 or lower. This is exactly what the parameter is for.
• Weird face mapping with a mismatched driver - try vox256_2Source and/or flip relative to true.
• Slow - yes. 6 minutes for the demo video is the honest number. Cut the driving video's frame count with VHS_LoadVideo's frame_load_cap if you just want to test.
• Missing keypoint error - module_fsrt/models/kp_detector.pt isn't there.