HY OmniWeaving Text Encode

OmniWeaving isn't just a video model with a fancy image-to-video mode - it's a reason-then-generate pipeline. A Qwen2.5-VL MLLM sits in front of the diffusion transformer, reads your prompt plus any reference frames, and produces the actual text conditioning the model generates from. HY OmniWeaving Text Encode is the node that runs that thinking brain inside ComfyUI. Where a normal CLIPTextEncode is a one-way trip from words to embeddings, this node is closer to a small agent: it decides what task you're doing, builds the right system prompt, optionally injects vision tokens for reference images, and can even let the model expand your prompt before encoding it.

How it works

The node is built around a task combo (t2v, i2v, interpolation, reference2v, editing, tiv2v) that maps to the original repo's prompt_mode and its tracked crop_start values - OmniWeaving chops the prompt at a task-specific point, and the node preserves that exactly. Each task also carries its own system prompt ("Describe the key features of the input image... then explain how the user's text instruction should alter the image..."), so the Qwen knows what kind of video to plan.

If use_visual_inputs is on (default), reference images become vision tokens - roughly 400 tokens each, capped by max_visual_inputs (default 8). The semantic_images input takes the VAE-roundtripped frames from HY OmniWeaving I2V Semantic Images - that's the text-side multimodal path that drives the current validated workflow. clip_vision_output (from Redux Vision Encode) and reference_images are the alternative visual routes.

The think options are the fun part. think=true turns on AR prompt expansion - the Qwen literally generates an enhanced version of your prompt first. Two modes:

• legacy_rewrite - the Qwen rewrites the prompt and it gets re-encoded once. This mirrors what the public OmniWeaving repo does, and it's the safer, closer-to-reference choice.
• merge_hidden - keeps your original prompt as the base branch and merges the AR-generated continuation's hidden states in as a second branch. It's the paper-style experiment for more motion, at the cost of prompt fidelity. It's stable now but still the pack's known quality risk.

think_keep_tokens (0 = keep the whole generated branch) and think_max_new_tokens (default 1000) tune the expansion. deepstack_layers (default "8,16,24") and setclip (default on) are deepstack/setclip hooks - leave them unless you're chasing the compat rabbit hole.

The inputs that matter

• clip - from HY OmniWeaving Text Encoder Loader (the dual Qwen + ByT5 clip, not a stock one).
• prompt - your actual prompt, multiline.
• task - match it to HY OmniWeaving Conditioning. Non-negotiable.
• think - start false; once the base path works, flip it on and try legacy_rewrite.

Output is one CONDITIONING, which feeds HY OmniWeaving Conditioning (positive and negative - run two of these nodes, one for each).

Installing it

Same pack, same install. ComfyUI Manager ("hy_omniweaving") or:

cd ComfyUI/custom_nodes
git clone https://github.com/Shiba-2-shiba/hy_omniweaving_comfyui_unofficial

Restart after. No pip dependencies; needs a recent ComfyUI for the extension API. The heavyweight downloads (7B Qwen, 8.3B fp8 diffusion model) are listed in the README.

Common issues

• Output looks unmoored from your prompt - check the task matches between this node and Conditioning, then make sure the OmniWeaving-finetuned Qwen (not a vanilla Qwen) is loaded.
• merge_hidden weakens prompt fidelity - known, documented, expected. The AR branch can over-describe static appearance. That's why legacy_rewrite is the default and the recommended starting point.
• Multilingual fragments or "str str str" in a rewrite - that was the old lm_head bug, fixed in current builds. If you still see it, update the pack.
• Slow? Yes. Community testing showed OmniWeaving wants 30–50 steps and CFG, and the whole thing crawls on consumer cards. The thinking pass adds time on top of sampling. Budget accordingly.