ComfyUI_FrameInterpolation

A ComfyUI custom node for 2× video frame interpolation using a Fourier-transform-based phase difference algorithm on a complex steerable pyramid.

---

Overview

Standard frame interpolation blends pixel intensities linearly between two frames, which produces blurry or ghosted results whenever objects are in motion. This node takes a different approach: instead of blending intensities, it interpolates the phase of image content at each spatial scale.

"Image structure is encoded primarily by phase. Interpolating the phase of each pixel produces a much higher quality result." — Phase-Based Frame Interpolation, exclav.es

The result is a synthesized intermediate frame that preserves sharp edges and fine structure with significantly less ghosting than naive linear blending.

---

Algorithm

1. Color Space Separation

Each input frame is converted from RGB to CIE Lab color space. The L (luminance) channel carries the structural information processed by phase interpolation. The a and b (chrominance) channels are handled with simple linear interpolation, since human vision is far less sensitive to chroma artifacts.

2. Complex Analytic Pyramid

A multi-scale complex pyramid is built from the luminance channel of each frame. At every pyramid level, a 2D analytic signal is computed via FFT by zeroing negative-frequency components (equivalent to a 2D Hilbert transform). This yields complex coefficients where:

• Amplitude encodes local contrast at that scale.
• Phase encodes the structural position of image features.

3. Phase Difference

For each corresponding pyramid level pair (pyr1[k], pyr2[k]), the per-pixel phase difference is computed as:

Δφ[k] = angle( pyr2[k] · conj(pyr1[k]) )

This gives the wrapped phase shift between the two frames at scale k, in the range (−π, π].

4. Hierarchical Phase Unwrapping

Raw phase differences are ambiguous modulo 2π. The unwrapping resolves this by propagating motion estimates coarse-to-fine:

1. The coarsest level is taken as-is (no ambiguity at low resolution).
2. For each finer level, the coarse unwrapped phase is upsampled and used as a prediction.
3. The residual between the fine difference and the coarse prediction is rounded to the nearest multiple of 2π and subtracted — effectively choosing the physically consistent branch.

5. Phase Consistency Adjustment

A final pass ensures each unwrapped value agrees with the raw phase difference modulo 2π, correcting any remaining inconsistencies introduced by the upsampling step.

6. Interpolation

For an interpolation factor α ∈ [0, 1] (midpoint = 0.5):

| Component | Formula | |-----------|---------| | Amplitude | (1 − α) · amp1 + α · amp2 (linear blend) | | Phase | φ1 + α · Δφ_unwrapped | | Complex coeff | amp_interp · exp(i · phase_interp) |

7. Reconstruction

The interpolated complex pyramid is collapsed back into a luminance image by summing real parts across scales. Chrominance channels are linearly blended. Both are then recombined into Lab and converted back to RGB.

8. Video Assembly

The output video contains the original frames interleaved with synthesized intermediate frames, doubling the frame count. The output frame rate is set to 2× input FPS so playback duration is unchanged.

---

Node Reference

Category: video/interpolation Display Name: Phase-Based Frame Interpolation (2x) Internal Name: PhaseBasedFrameInterpolation

Inputs

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | video_path | STRING | — | Absolute path to the input video file (mp4, avi, mov, etc.) | | pyramid_levels | INT (3–10) | 6 | Number of pyramid levels. More levels improve unwrapping accuracy at coarser scales but increase processing time. | | output_filename | STRING | phase_interp_output.mp4 | Filename for the output video, written to ComfyUI's output directory. |

Outputs

| Parameter | Type | Description | |-----------|------|-------------| | output_video_path | STRING | Absolute path to the interpolated output video. |

---

Installation

cd ComfyUI/custom_nodes
git clone <this-repo-url> ComfyUI_FrameInterpolation
pip install -r ComfyUI_FrameInterpolation/requirements.txt

Restart ComfyUI. The node will appear under video/interpolation.

---

Requirements

| Package | Purpose | |---------|---------| | numpy | Array operations, FFT | | scipy | (available for future extensions) | | opencv-python | Video I/O, color space conversion, pyramid resizing |

---

Performance Notes

• Processing is CPU-bound. A 1-minute 1080p clip at 30 fps may take several minutes depending on hardware.
• pyramid_levels = 6 balances quality and speed well for most content.
• Increasing pyramid_levels improves phase unwrapping for large motions but scales processing time roughly linearly.
• Memory usage scales with the total number of pixels × pyramid levels. Very high-resolution videos should be tested with caution.

---

Limitations

• The algorithm assumes small-to-moderate motion between frames. Large displacements (more than roughly a quarter of the image width per frame) may still produce artifacts, as the phase-unwrapping heuristic can fail at extreme shifts.
• Occlusions and dis-occlusions (objects appearing or disappearing) are not explicitly handled; these regions will blend toward the nearest available content.
• Only luminance is phase-interpolated; chroma is linearly blended, which may produce mild saturation artifacts on color edges under large motion.

---

References

• Phase-Based Frame Interpolation — A Julia Implementation
• Simoncelli, E. P., & Freeman, W. T. (1995). The steerable pyramid: A flexible architecture for multi-scale derivative computation. ICIP 1995.
• Fleet, D. J., & Jepson, A. D. (1990). Computation of component image velocity from local phase information. IJCV 5(1).