RunningHub VideoAsPrompt Sampler(CogVideoX)

Here's the pitch, in one line: you give this node an image, a reference video, and a sentence - and it generates a video where the image's subject moves with the reference video's motion. This is the RunningHub VideoAsPrompt Sampler(CogVideoX), the business end of the HM-RunningHub/ComfyUI_RH_VideoAsPrompt pack, which wraps ByteDance's Video-As-Prompt research on the CogVideoX-5B base.

If the concept sounds familiar, it should. Motion transfer is a crowded room in ComfyUI: Wan Animate, SCAIL-2, VACE all do a version of it. The difference here is that VAP doesn't extract a pose skeleton and re-render it. The reference video's latents get fed straight into the transformer as conditioning - a modified 3D rotary position embedding that lays the reference frames out alongside your generated frames, plus a second text prompt (prompt_mot_ref) that describes the reference motion. The video is the prompt. That makes it more semantic and less puppet-y than skeleton-driven approaches, and also less precise when you need exact choreography.

The workflow

Load Image → Load Video → [VideoAsPrompt Loader] → [this node] → Save Video

The Loader hands in the RH_VideoAsPrompt_Pipeline; the sampler takes over from there. Internally it resamples your reference video to exactly num_frames frames using even spacing, so a 4-second clip becomes a 49-frame guide whether your source was 24fps or 60fps. Then it runs the pipeline and hands you back an IMAGE tensor.

Inputs that actually matter

You set almost everything on this node, but four inputs carry the work:

• ref_video - your motion source, as an IMAGE frame sequence. Load it with a VideoHelperSuite (VHS) load-video node; a single static image in here does nothing useful.
• prompt - what the output video depicts. Normal text-to-video style prompting.
• prompt_mot_ref - the motion description for the reference clip ("camera pans right, subject walks forward"). This feeds the second conditioning stream; leave it empty and the model guesses the motion on its own.
• image - the start frame your output animates from. This is CogVideoX image-to-video, so the first frame of your result comes from here.

Then the boring but important ones: height/width default to 480×720 (portrait, 16–1024 range), num_frames defaults to 49, num_inference_steps defaults to 50. That's a roughly 6-second clip at 8fps. Fewer steps (25–35) gets you faster, slightly rougher motion; CogVideoX likes dynamic CFG, which this node enables automatically.

A trap worth knowing about

The seed input has a tooltip and everything - and the code reads it, comments it out, and never uses it. Changing the seed changes nothing. Don't burn an evening hunting for the "right" one; this is a known quirk of the pack, not a bug in your setup. Also note the sampler writes output.mp4 into ComfyUI's working directory on every run as a side effect, and returns the frames as an IMAGE for you to save properly. Ignore the stray file, save the tensor via VHS with your own fps.

Installing

Same story as its sibling node: ComfyUI Manager (search RH_VideoAsPrompt), or:

cd ComfyUI/custom_nodes/
git clone https://github.com/HM-RunningHub/ComfyUI_RH_VideoAsPrompt.git
cd ComfyUI_RH_VideoAsPrompt
pip install -r requirements.txt

Restart, then download the VAP-modified CogVideoX-5B weights into ComfyUI/models/Video-As-Prompt/CogVideoX-5B/{vae,transformer} - ~20GB, and don't substitute the vanilla THUDM release; the MOT transformer expects the VAP weights. Plan for ~12GB VRAM (INT8 + CPU offload are forced on) and a patient first run. If it errors on load, the model path is wrong; if it runs but motion looks weak, spend your effort on prompt_mot_ref before touching anything else.