CV Refine Flow (Variational)

cv2.VariationalRefinement is the energy-minimising polish that DIS runs internally - brightness constancy plus smoothness, solved iteratively. This node exposes it as its own step, which means you can point it at any dense flow field: a Farneback result, a DIS result with refinement turned off, a RLOF or TV-L1 field, or a field you built yourself by adding two others together.

The most common reason people reach for it: they ran DIS with refinement off because it was faster, or they're using a matcher that has no refinement step at all, and the field is holey and blocky around motion boundaries. This fixes that. What it will not do is find motion the input missed. It refines; it does not search. If the input flow is empty in a region, it stays empty.

Mechanism, honestly stated

It minimises an energy with three weighted terms, which is exactly what the inputs are:

• delta (5) - colour constancy. "Brightness in frame A should match the corresponding brightness in frame B." This is the term that does the actual matching work.
• gamma (10) - gradient constancy. This is what keeps the refinement sane through illumination changes, where brightness constancy alone would produce bogus motion.
• alpha (20) - smoothness. Raise it for a smoother, more filled-in field; lower it to keep sharp discontinuities at motion boundaries, which is what you want if you're trying to separate a moving subject from a static background.

iterations (5) is the outer fixed-point count - more is closer to the energy minimum and slower. Then the solver internals: sor_iterations (5) and omega (1.6), the successive-over-relaxation iterations per outer step and its relaxation factor. 1.6 is the standard fast-converging value; leave it unless you're doing something specific.

Inputs and outputs

frame_a and frame_b are the same two frames the flow was computed from - order matters, and they're converted to grayscale internally. They accept an IMAGE/MASK directly (frame 0 of a batch) or an NPARRAY, and frame_b is resized to frame_a if they differ. flow is the HxWx2 float32 field to refine, resized to the frames if it doesn't match.

That copy-then-refine behaviour is deliberate and worth noting: the upstream array's cached value is never mutated. In a cached graph, a node that edited its input in place would give you results that depend on execution order, which is a genuinely nasty class of bug.

Out come three things: flow, the refined HxWx2 (dx, dy) field; magnitude, the per-pixel motion length in pixels as HxW; and angle, the motion direction in radians, which is what CV Flow To Color expects in its default mode. Wire the refined flow into CV Draw Flow Grid for arrows, or into CV Flow Map if you want to apply the motion as a remap rather than just look at it.

Install

Part of ComfyUI CV (bmad4ever/comfyui_cv), GPL-3.0 fork of opencv-comfyui:

cd ComfyUI/custom_nodes
git clone https://github.com/bmad4ever/comfyui_cv
pip install "opencv-contrib-python-headless~=5.0.0.93"
# restart ComfyUI

Manager: search the pack title. Python ≥ 3.12 and a V3-API ComfyUI. The node exists because VariationalRefinement is a cv2 class, and the pack's ~470 generated wrappers only cover plain functions - so this is the only path to it from a graph in this pack.

Practical notes

Chain it, don't stack it. Running refinement twice is legal and the pack mentions it, but the second pass mostly smooths what the first pass produced. If you want a smoother field, raise alpha, not the pass count.

Grayscale means grayscale. Colour information in the frames is dropped, so distinguishing two same-luminance regions that move differently is not something this term can do.

It's CPU work over every pixel of a pair, per execution. This is not the place to build a 600-frame batch and hit run; test on two frames, then decide whether you need the whole clip.

The pack's workflows/39_optical_flow_playground.json is the clearest illustration: it runs DIS three ways on one frame pair - medium preset, the same with refinement off, and that raw blocky field pushed through this node - so the before/after sits side by side in one graph. If the blocky-vs-polished difference isn't visible on your footage, this node isn't going to transform your pipeline either.