Video Mask Propagator

Nobody wants to hand-mask 121 frames. The reason masking video stayed painful for so long is that masks don't move with the picture - an object drifts, stretches and rotates, and a mask you copy forward is wrong by frame three.

Video Mask Propagator is a rotoscoping helper: you supply a mask sequence where only some frames are drawn, plus optical flow vectors for the whole clip, and it fills the empty frames by warping the neighbouring drawn masks along that flow. Frames that already contain any mask content are treated as keyframes and passed through untouched.

The mechanism, and why it's picky about flow direction

Both inputs are tensors. masks is [B, H, W] - one mask per frame. flow_vectors is an [B, H, W, 3] IMAGE, with R as horizontal (U) and G as vertical (V) displacement, exactly what Radiance Optical Flow produces.

For each empty frame the node builds a normalised sampling grid, offsets it by the flow, and warps the previously-filled frame with a bilinear grid_sample on the GPU. Forward carries masks from earlier frames, Backward from later frames, and Bidirectional (the default) runs both and keeps the maximum - the union of the two warps on filled frames. That union behaviour is why bidirectional usually wins: a mask that the forward pass loses when an object rotates out of itself is often recovered from the other side.

The flow convention here is documented in the source and it is not the obvious one. flow_vectors[i] lives on frame i and points to where each pixel was in frame i-1; index 0 is zero. An earlier version read the vectors with the opposite sign and a half-pixel-off normalisation, so masks drifted the wrong way - an IoU of about 0.6 after one frame, and zero by frame three. If you're on an old build and your masks seem to slide sideways, that's what you're looking at.

Three inputs, one output. That's the whole node:

• masks (MASK) - the sequence, one per frame.
• flow_vectors (IMAGE) - 32-bit flow from Radiance Optical Flow, not an arbitrary RGB image.
• propagation_mode - Forward, Backward, Bidirectional.
• propagated_masks (MASK) - feeds inpainting, grades, relight, whatever the mask was for.

If the batch is a single frame the node returns it unchanged; there's nothing to propagate.

Install

ComfyUI Manager → Radiance → install → restart → refresh. Manual:

cd ComfyUI/custom_nodes
git clone https://github.com/fxtd-studios/radiance.git
cd radiance
python -m pip install -r requirements.txt

Windows portable users: run pip with python_embeded\python.exe. The pack pulls in OpenImageIO, OpenColorIO, OpenCV, imageio-ffmpeg and friends; install.py handles the two colour libraries. Nothing here downloads a model - just make sure Radiance Optical Flow is upstream in the graph, since flow_vectors wants its output and nothing else.

Where people get burned

Propagation is not tracking. Flow warps pixels; it doesn't know what a subject is. Occlusions - a hand passing in front of a face, a pole crossing a shot - produce smears that then become keyframes for the next round, because every frame with content is a keyframe. Fill fewer frames per pass and place deliberate keys at occlusions, or you'll be fixing artefacts with more artefacts.

Fast motion and cuts. Large inter-frame displacement breaks the warp, and a hard cut makes both directions meaningless. Cut the shot first (Radiance has scene-cut detection for exactly this) and propagate per shot.

You still need a mask source. The pack removed its own Roto node in 3.5.0 - its comment is blunt about it: draw masks with ComfyUI's mask editor or a SAM / matting node and propagate them from here. And note that Radiance's bundled SAM Loader and SAM Mask Generator are labelled placeholders that error when executed, so bring your own segmentation.

Masks that don't line up with your plate. The mask batch and the flow batch have to be the same length and the same resolution. There's no resizing and no padding - mismatched counts produce a short or wrong-length result rather than a friendly error.

Pair it with a matting or depth pass and this is the least glamorous, most time-saving node in a rotoscoping graph.