ComfyUI-YogurtKiwiEdit

中文 | English

一个用于在 ComfyUI 中调用 Kiwi-Edit Diffusers 模型进行视频编辑的自定义节点插件。当前版本只保留专家拆分流程，把 Kiwi-Edit 手动拆成 VAE 编码、MLLM 上下文编码、去噪生成和 VAE 解码 4 个阶段，便于分段控显存。

原始项目 / Original project: showlab/Kiwi-Edit

中文

✨ 功能

• 提供 5 个节点：
  • Load Kiwi Edit Pipeline
  • Kiwi Edit VAE Encode Inputs
  • Kiwi Edit MLLM Encode Context
  • Kiwi Edit Generate Latents
  • Kiwi Edit VAE Decode
• 直接对接 ComfyUI 原生 VIDEO 类型
• 支持从 models/KiwiEdit 目录中选择 Kiwi-Edit Diffusers 模型
• 支持 source_input_video、ref_image、input_video
• 专家链路会在每个阶段只保留当前所需模块在加速设备上

📦 安装

1. 将本插件放到：

ComfyUI/custom_nodes/ComfyUI-YogurtKiwiEdit

2. 安装依赖：

pip install -r custom_nodes/ComfyUI-YogurtKiwiEdit/requirements.txt

3. 重启 ComfyUI。

📁 模型放置

请将 Kiwi-Edit 的 Diffusers 模型目录放到：

ComfyUI/models/KiwiEdit/

目录结构示例：

ComfyUI/
  models/
    KiwiEdit/
      kiwi-edit-5b-instruct-reference-diffusers/
        model_index.json
        transformer/
        vae/
        scheduler/
        ...

说明：

• 插件会按 ComfyUI 的方式注册 models/KiwiEdit
• 下拉框会扫描所有包含 model_index.json 的子目录
• 如果界面中显示 [No models found in models/KiwiEdit]，说明当前目录下没有检测到可用模型

🗂️ Diffusers 模型下载

| 模型 | 类型 | Hugging Face | | --- | --- | --- | | kiwi-edit-5b-instruct-only-diffusers | 仅在 Instruction Data 上微调 | linyq/kiwi-edit-5b-instruct-only-diffusers | | kiwi-edit-5b-reference-only-diffusers | 仅在 Reference Data 上微调 | linyq/kiwi-edit-5b-reference-only-diffusers | | kiwi-edit-5b-instruct-reference-diffusers | 同时在 Instruction 和 Reference Data 上微调 | linyq/kiwi-edit-5b-instruct-reference-diffusers |

🧩 节点说明

1. Load Kiwi Edit Pipeline

输入：

• model_name
• dtype
• cpu_offload

输出：

• pipeline

2. Kiwi Edit VAE Encode Inputs

用途：预处理并编码所有 VAE 侧输入。

输入：

• pipeline
• source_video
• source_input_video，可选
• ref_image，可选
• input_video，可选
• max_frames
• max_pixels
• tiled

输出：

• conditioning

行为：

• source_input_video 未连接时会回退到 source_video
• ref_image 会把当前 batch 中所有图都转成参考图列表
• input_video 只会先编码成 latent，不会在这里加噪
• tiled 现在会实际控制 diffusers pipe.vae.enable_tiling()/disable_tiling() 的编码路径

3. Kiwi Edit MLLM Encode Context

用途：只做 Qwen/MLLM 上下文编码。

输入：

• pipeline
• conditioning
• prompt

输出：

• context

4. Kiwi Edit Generate Latents

用途：只做 scheduler、噪声初始化和 denoising loop。

输入：

• pipeline
• conditioning
• context
• num_inference_steps
• guidance_scale
• seed
• sigma_shift
• denoising_strength

输出：

• latents

说明：

• 当前版本不实现 negative_prompt
• guidance_scale 保留为接口参数，但按上游 Kiwi 当前实现不会触发 CFG 路径
• tile_size/tile_stride 暂不暴露，因为上游 pipeline 当前没有真正使用它们

5. Kiwi Edit VAE Decode

用途：只做最终 VAE 解码。

输入：

• pipeline
• latents
• tiled

输出：

• video

说明：

• tiled 会实际控制 diffusers pipe.vae.enable_tiling()/disable_tiling() 的解码路径
• 输出会保留原视频的 fps 和音频
• metadata 会继续透传，但是否完整 round-trip 取决于当前 ComfyUI 的 VIDEO 实现

🔗 推荐工作流

Load Kiwi Edit Pipeline -> pipeline
LoadVideo -> Kiwi Edit VAE Encode Inputs(source_video)
pipeline -> Kiwi Edit VAE Encode Inputs
Kiwi Edit VAE Encode Inputs -> Kiwi Edit MLLM Encode Context
pipeline -> Kiwi Edit MLLM Encode Context
Kiwi Edit MLLM Encode Context -> Kiwi Edit Generate Latents
Kiwi Edit VAE Encode Inputs -> Kiwi Edit Generate Latents
pipeline -> Kiwi Edit Generate Latents
Kiwi Edit Generate Latents -> Kiwi Edit VAE Decode
pipeline -> Kiwi Edit VAE Decode
Kiwi Edit VAE Decode -> SaveVideo

如果需要独立的 VAE conditioning 视频：

LoadVideo(source)
  -> Kiwi Edit VAE Encode Inputs.source_video

LoadVideo(source_input)
  -> Kiwi Edit VAE Encode Inputs.source_input_video

如果需要 video-to-video 输入：

LoadVideo(input_video)
  -> Kiwi Edit VAE Encode Inputs.input_video

⚠️ 注意事项

• 本插件要求模型是 Diffusers 目录，不是单个 .safetensors
• negative_prompt 暂不提供
• guidance_scale 目前仅保留接口，不会触发上游 CFG
• tile_size/tile_stride 暂不暴露

---

English

✨ Features

• Provides 5 nodes:
  • Load Kiwi Edit Pipeline
  • Kiwi Edit VAE Encode Inputs
  • Kiwi Edit MLLM Encode Context
  • Kiwi Edit Generate Latents
  • Kiwi Edit VAE Decode
• Works directly with ComfyUI native VIDEO type
• Selects Kiwi-Edit Diffusers models from models/KiwiEdit
• Supports source_input_video, ref_image, and input_video
• Keeps only the currently needed module group on the accelerator for each stage

📦 Installation

1. Place this plugin in:

ComfyUI/custom_nodes/ComfyUI-YogurtKiwiEdit

2. Install dependencies:

pip install -r custom_nodes/ComfyUI-YogurtKiwiEdit/requirements.txt

3. Restart ComfyUI.

📁 Model Placement

Put your Kiwi-Edit Diffusers model folders under:

ComfyUI/models/KiwiEdit/

The plugin scans subfolders containing model_index.json and exposes them in the model_name dropdown.

🗂️ Diffusers Model Zoo

| Model | Type | Hugging Face | | --- | --- | --- | | kiwi-edit-5b-instruct-only-diffusers | Fine-tuned only on instruction data | linyq/kiwi-edit-5b-instruct-only-diffusers | | kiwi-edit-5b-reference-only-diffusers | Fine-tuned only on reference data | linyq/kiwi-edit-5b-reference-only-diffusers | | kiwi-edit-5b-instruct-reference-diffusers | Fine-tuned on both instruction and reference data | linyq/kiwi-edit-5b-instruct-reference-diffusers |

🧩 Nodes

1. Load Kiwi Edit Pipeline

Loads the diffusers pipeline and returns a reusable pipeline handle.

2. Kiwi Edit VAE Encode Inputs

Preprocesses and encodes VAE-side inputs.

• Falls back from source_input_video to source_video when not connected
• Converts the full connected ref_image batch into a reference-image list
• Encodes input_video into latents, but does not add noise yet
• tiled now actually controls diffusers pipe.vae.enable_tiling()/disable_tiling() for VAE encode

3. Kiwi Edit MLLM Encode Context

Runs only the Qwen/MLLM context encoder.

4. Kiwi Edit Generate Latents

Runs scheduler setup, noise initialization, and the denoising loop.

• negative_prompt is not exposed in this version
• guidance_scale is kept for interface compatibility, but does not activate CFG with the current upstream Kiwi implementation
• tile_size/tile_stride are not exposed because the current upstream pipeline does not actually use them

5. Kiwi Edit VAE Decode

Runs only the final VAE decode and returns VIDEO.

• Requires both pipeline and latents
• tiled now actually controls diffusers pipe.vae.enable_tiling()/disable_tiling() for VAE decode
• Preserves original fps and audio
• Metadata is still passed through, but round-trip retention depends on the current ComfyUI VIDEO implementation

🔗 Recommended Workflow

Load Kiwi Edit Pipeline -> pipeline
LoadVideo -> Kiwi Edit VAE Encode Inputs(source_video)
pipeline -> Kiwi Edit VAE Encode Inputs
Kiwi Edit VAE Encode Inputs -> Kiwi Edit MLLM Encode Context
pipeline -> Kiwi Edit MLLM Encode Context
Kiwi Edit MLLM Encode Context -> Kiwi Edit Generate Latents
Kiwi Edit VAE Encode Inputs -> Kiwi Edit Generate Latents
pipeline -> Kiwi Edit Generate Latents
Kiwi Edit Generate Latents -> Kiwi Edit VAE Decode
pipeline -> Kiwi Edit VAE Decode
Kiwi Edit VAE Decode -> SaveVideo