MotionMaskFromDepth

A video of a mostly-static scene with one person walking through it has two kinds of geometry: the background you can treat as one solid world, and the moving bits you have to model separately. MotionMaskFromDepth is the pack's detector for that split - it looks at a depth+pose sequence and flags every pixel that's moving, outputting a mask where 1.0 means "this pixel is dynamic." It's a required early step in the pack's video-to-4D-world pipeline, sitting between pose estimation and everything downstream.

Why depth-warp residual and not an optical flow model? Because it fits the pipeline. The pack already estimates per-frame camera poses (VideoPoseEstimator) and depth maps. If the world were static, you could take frame t's depth, reproject it through the pose change into frame t+gap, and get exactly the measured depth there. Wherever that prediction disagrees with reality, something in the scene moved. That disagreement - the depth residual - is your motion signal, no extra network required. The README's TODO list even flags SAM2-based refinement as a future improvement, which tells you the current masks come purely from this depth-warp logic.

How it works

For each frame it unprojects the depth into 3D points, transforms them by the camera pose into the world, then reprojects into neighboring frames (forward and backward by frame_gap, default 4). It samples the neighbor's actual depth at those projected positions and computes the relative residual |d_proj − d_sampled| / d_sampled. Any pixel whose residual exceeds threshold (default 0.1) is flagged dynamic. The final mask gets dilated by dilate pixels (default 2) so you don't get thin, torn edges around moving objects.

The inputs: depth_seq ([T,H,W] radial distance - note: ray depth, not Z-depth), trajectory ([T,4,4] world-to-camera poses; a shorter [K,4,4] path is interpolated up to T), input_projection and input_horizontal_fov, then threshold/frame_gap/dilate, and the usual device picker. One output: motion_mask, a MASK (1.0 = moving).

Tuning it

• threshold - lower (0.05) catches more motion but also depth-estimation noise; higher (0.15+) keeps only strong motion. Start at 0.1.
• frame_gap - larger gaps make small/slow motion easier to catch, at the cost of more reprojection error on static scenes.
• dilate - bump it if moving objects' edges come out with halo artifacts where static pixels leak in.

Where it fits

In video_to_4d_world.json: VideoPoseEstimator → MotionMaskFromDepth → VideoToFusedSplats (which keeps only static pixels for the fused world) and → EstimateTracks (which tracks the dynamic pixels for the 4D Gaussians). Get the mask wrong and you either fuse the walker into your "static" background or track the background as motion. Watch it first on a few frames.

Installing it

Part of camera-comfyUI. Manager → Custom Nodes Manager → camera-comfyUI → Install → restart, or:

cd ComfyUI/custom_nodes
git clone https://github.com/Alexankharin/camera-comfyUI.git
cd camera-comfyUI && python install.py

MotionMaskFromDepth itself needs no optional deps, but feeding it requires VideoPoseEstimator (VGGT), which install.py pulls in - the facebook/VGGT-1B weights (~5 GB) download on first use.

Common issues

• Mask lights up the whole background - depth noise at textureless areas is being read as motion. Raise the threshold, or check your depth is ray depth (Z-depth input bows geometry and inflates residuals).
• Moving object barely masked - lower the threshold or widen frame_gap; slow motion is invisible at gap 4 if it moves less than a pixel.
• Everything is static except a blob - that's often a depth discontinuity (object boundary) rather than motion; dilate will inflate it, so keep dilation modest if you see this.