PuppetFace ▸ Landmark + Cursor Overlay

If you've seen those "manipulating an AI talking head like a puppet" demo clips - the ones where a mouse cursor with a little grab ring hovers over a face mesh and drags landmarks around - you've seen exactly what PuppetFaceOverlay (display "PuppetFace ▸ Landmark + Cursor Overlay") draws. It takes a batch of frames of a face and renders landmark dots, an optional connecting mesh, and an animated cursor that tracks a chosen facial landmark. It's the graphics layer for that puppet-rig presentation look, built into the MSCH Nodes pack.

What it does and doesn't do

Read the name carefully: this is an overlay. It does not animate the person's face, does not move their mouth, does not puppet anything - it draws tracking graphics over frames you already have. You bring the video of a face (generated singing, a talking avatar, real footage), and PuppetFace adds the landmark visualization and a cursor that tracks the face across the shot, as if someone were operating it. If the actual "puppeteering" appeal is what you're after, that happens upstream or in an editor; this node is the presentation layer on top.

How the tracking works

Landmark sourcing is automatic, in priority order: an ONNX SCRFD + 2d106det tracker (via onnxruntime) when the models are present - it sticks to a deforming face - then MediaPipe FaceMesh (478 points, refine_landmarks on, so iris points included) if the ONNX models aren't there, then a plain OpenCV Haar-cascade oval as a zero-dependency last resort. The 0–477 landmark index range tells you the numbering follows the FaceMesh convention. Because the fallbacks are built in, the node runs even with nothing optional installed - it just degrades from precise tracking to an approximate face box, and prints a hint in the console about pip-installing mediapipe for real tracking.

show_dots toggles the landmark dots (with dot_set choosing key points, contours, tesselation, or all 478), dot_radius/dot_color/dot_opacity style them, and show_mesh draws the connecting web at mesh_opacity. The cursor is its own track: cursor_mode of track_landmark follows a target (nose, mouth, eye_left, eye_right, chin, or a raw landmark_index via cursor_target), sweep glides it between the two normalized sweep_start/end coordinates, and off hides it. cursor_scale sizes it and grab_ring draws the little ring that sells the "grabbing" gesture.

Wiring it

Input is images (an IMAGE batch of face frames), output is images (the same frames with graphics drawn). Chain it between your face video source and an encoder: Load Video → PuppetFaceOverlay → PuppetFaceSaveVideo (the pack's OpenCV MP4 saver), and keep the source's fps flowing through so the cursor tracks at the right speed. If you're using the pack's own PuppetFaceLoadVideo, connect its fps output.

Install

Part of the unified MSCH Nodes pack:

cd ComfyUI/custom_nodes
git clone https://github.com/mariobilly/msch-comfyui-nodes.git
cd msch-comfyui-nodes
python -m pip install -r requirements.txt

(ComfyUI Manager: search "MSCH Nodes" / msch-comfyui-nodes, restart.) For the precise ONNX tracking you additionally need onnxruntime and the separately supplied face models - the README marks that optional; without it the approximate fallback still works, so this node never hard-fails on a missing dependency.

Gotchas

Biggest misunderstanding is expecting it to move the face - it won't; if your source has no head motion, the cursor just sits on a static landmark and the whole effect looks dead, so feed footage with some movement. Second, if dots look like a loose oval instead of real landmarks, you're on the Haar fallback: install the ONNX models (or mediapipe) for actual tracking. And mismatch your fps and the cursor will lag or skate ahead - keep frame rate consistent through the chain.