OpenSora Encode Reference

OpenSora v3 does image-to-video, and this is the node that makes it happen. You feed it a single image, it encodes that image into the latent space OpenSora works in, and it builds the instructions that tell the sampler to anchor your video to that frame. Without it, you're stuck with pure text-to-video; with it, you can turn a still into a moving scene.

How it works

The mechanism is worth understanding because it explains the edit_rate slider, which confuses everyone at first. The node normalizes your image into the [-1, 1] range the OpenSora VAE expects, encodes it with the same VAE the sampler uses, and then constructs a "mask strategy" - a compact string like 0,0,0,0,<frame_count>,<edit_rate>. That string is OpenSora's native syntax for reference-conditioning, and the sampler reads it at sampling time: it literally overwrites the first (or last) frames of the noise latent with your encoded reference, then sets how strongly those frames constrain the ones around them.

That last number is the edit_rate. At 0, the reference frame is locked in completely - the video starts as a perfect copy of your image and animates away from it. Higher values let the sampler edit the reference itself, so it's free to drift and change. Most people land between 0.1 and 0.3 for "animates from my image" and reserve high values for a stylized reinterpretation.

Inputs

• opendit_vae - the VAE loader's output. Same VAE as the sampler uses, so grab it from either place.
• ref_image (IMAGE) - the still you want to animate. Any image, but a clean, single-subject composition gives the strongest anchor.
• target_frame_start - first or last. Whether your reference pins the start of the clip or the end. last is your "video converges on this image" trick, useful for looping into a target pose.
• edit_rate - 0–1, default 0. As above: how much the sampler may alter the reference frames.

Output

The single output, opendit_ref (type OPENDITREF), is a small package: the encoded reference latent plus the mask strategy. It plugs into the optional opendit_ref input on OpenDiTConditioning - that's the only place it goes. If you forget to wire it there, the node does nothing, which is the most common "why is my image ignored" failure. The conditioning node carries the reference through to the sampler, which applies the mask when it seeds the latent.

Installing

Search "ComfyUI-OpenDiTWrapper" in ComfyUI Manager, or:

cd ComfyUI/custom_nodes
git clone https://github.com/kijai/ComfyUI-OpenDiTWrapper
pip install -r requirements.txt
pip install xformers --no-deps

Restart ComfyUI afterward; Windows portable users run pip via python_embeded\python.exe.

Gotchas

Because the reference is pasted into the latent as raw frames, your reference image's aspect ratio matters - OpenSora will stretch it to the sampler's width/height, so keep the image close to your target resolution to avoid distortion. And remember this node is where image-to-video gets its start frame, so the sampler's num_frames still controls clip length; edit_rate only controls how long the anchor holds.