VFX-grade 2D tracking for ComfyUI. Mocha/Nuke-style planar tracking, stabilize → process → destabilize round-trips, corner-pin compositing, and mask propagation — built on classical computer vision (OpenCV), so it runs fast on CPU. No GPU needed.

Why

AI video workflows constantly need tracking: propagate a mask for inpainting, lock a face for replacement, pin an insert onto a screen. Dedicated trackers (Mocha, Nuke) are expensive and live outside your graph. PlateTrack puts a production-style tracker directly into ComfyUI.

Nodes

| Node | What it does | |---|---| | Planar Track | Track a textured planar region (wall, screen, sign, face). Outputs TRACK_DATA, an annotated preview, and the ROI as a propagated mask. | | Point Track (1-4) | Track 1-4 explicit points; fits translation/similarity/affine/perspective. | | Track Stabilize | Lock the tracked region in place (crop_roi gives a tight crop — ideal img2img payload). | | Track Destabilize | Reapply the original motion to processed frames and composite over the plate. | | Corner Pin Composite | Warp an insert (image or per-frame batch) onto the tracked plane, supersampled. | | Track to Mask | Propagate any reference-frame mask along the track (planar warp). | | Mask Propagate | Propagate a reference mask in planar (rigid warp) or grid (deformation-aware) mode, with per-frame confidence. | | Mask Propagate (Cutie) | Deep video-object-segmentation propagation via Cutie (optional dependency). | | Face Track | Detect a face (YuNet, in core OpenCV) and output a similarity TRACK_DATA + a per-frame face mask. | | Track to Transform | Per-frame x/y/rotation/scale lists to drive other nodes. | | Export / Import Track | Canonical .platetrack.json, plus Nuke CornerPin2D .nk and After Effects Corner Pin keyframe text. |

The killer workflow: stabilized cleanup

Load Video ─┬─> Planar Track ──> Track Stabilize (crop_roi) ──> [inpaint / img2img] ──┐
            └──────────────────────────────────────────────> Track Destabilize <─────┘

Track a region, stabilize it into a motionless crop, run any generative process on it, then destabilize: the result is composited back into the moving shot. The round-trip is exact by construction (both directions use the same stored matrix).

Install

Via ComfyUI-Manager / the registry: search for PlateTrack, or comfy node registry-install comfyui-platetrack.

Manually:

cd ComfyUI/custom_nodes
git clone https://github.com/edgarciavfx/comfyui-platetrack
pip install -r comfyui-platetrack/requirements.txt

If you already have opencv-python (some node packs ship it), skip the pip install — do not install both opencv-python and opencv-python-headless.

Tracking quality

Pyramidal Lucas-Kanade with forward-backward validation, RANSAC/MAGSAC model fitting (translation / similarity / affine / homography).
Every frame is ECC-refined against the reference frame, which kills the drift that plain frame-to-frame chaining accumulates.
Per-frame confidence output; the preview quad turns yellow below 0.6 and red below 0.3 so drift is findable at a glance.
Keyframe pins: pinned corners are honored exactly, blended in smoothly, and corrections persist forward like a Mocha manual keyframe.

Verified against synthetic ground truth: <0.5 px corner RMS on clean plates, <1 px closure over a 240-frame loop, >40 dB stabilize round-trip PSNR.

Mask propagation

Three ways to carry a mask through a shot, cheapest first:

Track to Mask / Mask Propagate planar — warp the reference mask by the tracked homography. Exact for a rigid plane (a wall, a sign, a screen), one matrix per frame, effectively free.
Mask Propagate grid — for subjects that aren't flat (a face, cloth, a bag). A 6×6 lattice is LK-tracked from the reference frame and blended into a smooth per-cell field, so the mask bends with the surface. Needs the source images; where optical flow drops out it falls back to the planar homography, so it never does worse than planar on a rigid plane. Outputs per-frame confidence alongside the masks.
Mask Propagate (Cutie) — deep video object segmentation for hard cases (occlusion, big appearance change). Optional dependency; see below.

Face replacement

Face Track detects a face with OpenCV's YuNet — bundled in core OpenCV, so nothing extra to install — and fits a similarity track (translation, rotation, uniform scale) from the stable landmarks (eyes + nose), plus a per-frame face mask. It drops straight into the stabilized round-trip:

Load Video ─> Face Track ─┬─> Track Stabilize (crop_roi) ─> [face gen / img2img] ─┐
                          ├──────────────────────────────> Track Destabilize <────┘
                          └── face_mask ────────────────────────────────────────> (feather)

The model auto-downloads to models/platetrack/ on first run. See workflows/face_replace.json. Frames where detection momentarily drops out hold the nearest solved transform. An optional mediapipe backend is used if it's installed and selected (backend = mediapipe); YuNet is the default.

Optional dependencies

The core pack needs only OpenCV. Two nodes light up with extras:

pip install -r requirements-optional.txt   # mediapipe (face) 
# Cutie is installed from source:
pip install git+https://github.com/hkchengrex/Cutie.git

Cutie (deep VOS) powers Mask Propagate (Cutie). Pure PyTorch, runs on CPU (~1–3 s/frame at 480p); weights auto-download on first use.
mediapipe is an alternate Face Track backend. Its wheels don't yet cover Python 3.14, which is why YuNet is the default.

Taking tracks to Nuke / After Effects

Tracked in ComfyUI, finishing in a compositor? Export Track writes both:

nuke_cornerpin — a complete, animated CornerPin2D node as .nk text. Open the file (or the node's snippet output), copy, and paste straight into the Nuke DAG. start_frame sets where the curves begin (Nuke scripts are usually 1-based). The Y-flip to Nuke's Y-up coordinates is already applied, and from1-4 are set to the reference-frame corners, so the pin is an exact match-move of the tracked quad. Works in Nuke Non-Commercial.
ae_cornerpin — Adobe "8.0 Keyframe Data" text for the Corner Pin effect. Apply Corner Pin to your insert layer, copy the exported text to the clipboard, select the layer, and Paste. Keyframes are 0-based and per-frame; fps only fills the header, so match it to your comp's frame rate.

Both formats bake one key per frame from the same H_to_ref chain the apply nodes use — what you pin in the editor is exactly what lands in the compositor.

Correcting a track — the editor widget

Planar Track carries an editor right on the node. After you Queue once:

Scrub the tracked result on the timeline (drag the playhead, use ◀ ▶, or wheel over the strip). The confidence graph plots per-frame confidence with a dashed 0.6 guide, and the heat-strip below it colors the same data — green ≥ 0.6, yellow ≥ 0.3, red below — so drift is one glance away. The blue tick is the reference frame; diamonds are your keyframes. ⚠ Worst jumps to the lowest-confidence frame; click it again to tour the next-worst (already pinned frames are skipped).
Pin a bad frame by dragging any of the four corner handles to where they belong. That writes a keyframe (white quad) and dirties the node.
Re-queue. The tracker re-runs honoring the pin exactly and blending the correction into the surrounding frames, and the fix persists forward. That's the whole loop — re-track = re-queue. Delete pin removes a keyframe.

State lives in the node's track_ui STRING (hidden behind the editor), so it saves inside the workflow JSON and restores on reload. You can still hand-author it if you prefer — corner order is TL, TR, BR, BL in plate pixels:

{"keyframes": {"143": [[512,288],[768,290],[770,470],[510,468]]}}

Tests

The tracking core is pure numpy+OpenCV (no ComfyUI/torch imports):

pytest tests/            # accuracy, drift, round-trip, mask (planar+grid), face, export
python scripts/run_headless.py   # end-to-end against a headless ComfyUI --cpu

The face maths (similarity fit, gap-fill, hull masks) is tested against synthetic landmark motion; a network-guarded test also downloads YuNet and runs the detector. The headless run exercises grid mask propagation inside a real ComfyUI graph.