RunningHub VideoAsPrompt Loader

This node has a single input, type, and the dropdown contains exactly one option: CogVideoX. That's it. The whole job of the RunningHub VideoAsPrompt Loader is to load a ~5B-parameter video model into memory without turning your GPU into a space heater, then hand the result to its partner node. It's the boring front half of a two-node pack, and boring is fine - the interesting part is what it does on load.

The pack wraps ByteDance's Video-As-Prompt (VAP) project: given a reference image and a reference video, it animates the image using the motion of the video as the driving signal. Where tools like SCAIL-2 or Wan Animate extract pose skeletons to drive movement, VAP treats the reference video itself as the conditioning - the video is the prompt, plus a short text line describing its motion. This is the CogVideoX-5B variant, which is a slightly odd choice in 2026 when Wan has been the local video standard for a year. Blame the research lineage: CogVideoX was Z.ai's pre-Wan workhorse, and ByteDance's VAP release shipped variants on both.

What "load" actually does

Open the source and the load step reads like a checklist for running big video models on a small card:

Loads vae/ and transformer/ from ComfyUI/models/Video-As-Prompt/CogVideoX-5B in bfloat16
Enables VAE slicing and tiling to cut memory during encode/decode
Quantizes the transformer to INT8 with optimum-quanto and freezes it
Calls enable_model_cpu_offload() so weights shuffle to system RAM between layers

That last one is why the README claims ~12GB VRAM instead of the ~20GB a raw 5B model wants. The tradeoff is speed - CPU offload makes every step slower, so your first run (model load plus offload warm-up) will feel glacial. That's normal, not a hang.

Inputs and outputs

There's nothing to tune. type picks the pipeline variant and "CogVideoX" is currently the only one shipped, so just leave it. The output is a single RH_VideoAsPrompt_Pipeline object - wire that into the pipeline input of the RunningHub VideoAsPrompt Sampler(CogVideoX) node. That's the entire graph.

Installing it

The pack is HM-RunningHub/ComfyUI_RH_VideoAsPrompt, from RunningHub's GitHub org. Despite the name, it's fully local - no API key, no cloud call, which surprises people who know RunningHub as the hosted ComfyUI platform. Two install routes:

# ComfyUI Manager: search "RH_VideoAsPrompt", or:
cd ComfyUI/custom_nodes/
git clone https://github.com/HM-RunningHub/ComfyUI_RH_VideoAsPrompt.git
cd ComfyUI_RH_VideoAsPrompt
pip install -r requirements.txt

Then restart ComfyUI. requirements.txt is the real dependency list - diffusers>=0.30.0, transformers>=4.44.0, accelerate, and optimum-quanto>=0.2.0 for the INT8 quantization. If you use Manager, double-check the quanto install actually landed; that's the one that silently breaks on fresh setups.

The model and the gotchas

The model is the part most people trip on. It must live at exactly:

ComfyUI/models/Video-As-Prompt/CogVideoX-5B/
├── vae/
└── transformer/

The README points at the VAP dataset and the ByteDance weights collection, which makes sense - these are the VAP-modified CogVideoX-5B weights, not the vanilla THUDM release. Count on ~20GB of storage. If the Loader throws a "no such file or directory" style error, the model path is your suspect, not the node.

Two honest caveats. First, INT8 on the transformer saves VRAM but is a quality tax; you'll notice it more on detailed motion than on static frames, and there's no fp16 toggle in this pack if you have the VRAM. Second, the one-option dropdown is a hint this pack hard-codes CogVideoX today - if a Wan-based VAP variant ever ships, you'd see it here. Until then, set type, wire the pipeline forward, and go make something move.