ComfyUI Video-As-Prompt Node

A custom node for ComfyUI that integrates Video-As-Prompt for motion-guided video generation from image inputs.

✨ Features

• 🎬 Motion-Guided Generation: Use reference videos to control motion in generated videos
• 🖼️ Image-to-Video: Generate videos from image with reference motion guidance
• ⚙️ Memory Optimization: INT8 quantization + CPU offload for efficient inference
• 🚀 CogVideoX-5B: Based on powerful CogVideoX-5B model

🔧 Node List

• RunningHub VideoAsPrompt Loader: Load and initialize Video-As-Prompt pipeline
• RunningHub VideoAsPrompt Sampler(CogVideoX): Generate video from image with reference motion

🚀 Quick Installation

Step 1: Install the Node

# Navigate to ComfyUI custom_nodes directory
cd ComfyUI/custom_nodes/

# Clone the repository
git clone https://github.com/HM-RunningHub/ComfyUI_RH_VideoAsPrompt.git

cd ComfyUI_RH_VideoAsPrompt

# Install dependencies
pip install -r requirements.txt

Step 2: Download Required Models

Download the CogVideoX-5B model and place it in the following structure:

ComfyUI/models/Video-As-Prompt/
└── CogVideoX-5B/
    ├── vae/
    ├── transformer/
    └── ...

You can download from Video-As-Prompt Dataset or use the pretrained CogVideoX-5B model.

Step 3: Restart ComfyUI

📖 Usage

Basic Workflow

[Load Image] → [Load Video] → [RunningHub VideoAsPrompt Loader] → [RunningHub VideoAsPrompt Sampler] → [Save Video]

Generation Parameters

• image: Input image for video generation
• ref_video: Reference video for motion guidance
• prompt: Text description for the output video
• prompt_mot_ref: Text description for the reference motion
• height/width: Output video dimensions (default: 480x720)
• num_frames: Number of frames to generate (default: 49)
• num_inference_steps: Denoising steps (default: 50)

🛠️ Technical Requirements

• GPU: 12GB+ VRAM (with INT8 quantization + CPU offload)
• RAM: 16GB+ recommended
• Storage: ~20GB for CogVideoX-5B model
• CUDA: Required for optimal performance

⚠️ Important Notes

• Model Paths: Models must be placed in ComfyUI/models/Video-As-Prompt/ directory
• Memory Optimization: INT8 quantization and CPU offload are enabled by default for memory efficiency
• All model files must be downloaded before first use

🔗 References

• Video-As-Prompt Project
• Video-As-Prompt Dataset
• ComfyUI

📄 License

This project is based on the Video-As-Prompt project.

⭐ Citation

If you find this project useful, please consider citing the original Video-As-Prompt paper.

---

Developed by HM-RunningHub