智能视频裁剪 (Smart Video Crop)

Motion-transfer models care about the person, not the scenery. A wide shot where your dancer occupies a fifth of the frame produces a generation where the character is tiny and the pose signal is weak - SCAIL-2 and WanAnimate both work far better when the subject fills the frame. SmartVideoCrop is the automatic version of the manual "crop the clip to the person" step: it takes video frames plus SAM3 tracking data, finds the tracked subject's bounding region every frame, and outputs a smoothly following crop at whatever resolution you set.

It's built on a pretty clean mechanism. The node unpacks the packed masks from a SAM3 tracking result (SAM3_TRACK_DATA, the same driving_track_data you'd feed a SCAIL-2 node), combines the masks for the object IDs you select, computes the centroid of that mask per frame, and slides a crop window around it. The window is smoothed frame to frame so it doesn't jitter, and it watches for scene cuts: if the centroid jumps more than scene_cut_threshold between consecutive masked frames, it treats it as a cut and resets the smoothing instead of trying to glide across a hard transition.

The inputs that matter

• frames - the current chunk's video frames (B, H, W, C) (required).
• track_data - SAM3_VideoTrack output. This is the required input that makes everything else possible.
• object_ids - which tracked person(s) to follow, comma-separated like 1 or 1,2 (default 1). If SAM3 assigned your dancer ID 3, this is where you tell it.
• output_width / output_height - the crop resolution (defaults 480×864, both stepped to multiples of 16 for video-model friendliness).
• smoothing (default 0.3) - inter-frame smoothing: 0 is no smoothing (raw follow), 1 is completely static. Low values track fast but jitter; higher values are steadier but lag quick moves.
• mask_fallback (hold_last/center) - what happens when a frame's mask is empty (tracking lost the subject). hold_last keeps the last valid position; center returns to the middle of the frame.
• scene_cut_threshold (default 0.3) - normalized centroid-jump threshold for cut detection, 0 disables it.
• mask_frames (optional) - a second image stream (like a mask video) cropped with the exact same window, so masks and frames stay pixel-aligned after cropping.

Outputs are cropped_frames, cropped_mask (both cropped with the same window), and debug_info - a string you can route to a text preview to see what the tracker was doing.

Notes and gotchas

You need SAM3 tracking running upstream (an external tracking model; this node only consumes its output), and the tracking quality is the ceiling on crop quality - if SAM3 loses the subject, the fallback behavior kicks in. The default is a portrait 480×864, which suits a standing person; for dance where the subject moves wide, you may want to let output_width grow. Start with hold_last for fallback and only switch to center if tracking drops make the crop wander.

Install

Part of ComfyUI-CustomNodeKit. ComfyUI Manager → search "ComfyUI-CustomNodeKit", or:

cd ComfyUI/custom_nodes
git clone https://github.com/user2318/ComfyUI-CustomNodeKit.git
cd ComfyUI-CustomNodeKit
pip install -r requirements.txt

Restart ComfyUI. No models bundled - SAM3 itself is a separate install.