HYWorld2 QwenVL

Underneath the whole HY-World2 expansion pipeline there's a vision-language model deciding what the scene actually is. HYWorld2 QwenVL is that model exposed as a node. It loads a Qwen3-VL model and runs one of three jobs: caption what objects are in the scene, caption what a given trajectory is looking at, or refine your prompt. The output - a string plus an HYWORLD2_LLM_CONTEXT - is what the world-expansion stages use to keep generation aligned with reality instead of just hallucinating.

You don't strictly need to touch this node yourself - Trajectories and World Expansion drive QwenVL internally. But it's exposed for a reason: you can run it standalone to caption a trajectory set or a batch of images, inspect what the pipeline "thinks" it's looking at, and even fix bad captions before they propagate. It's the honest, visible brain of the operation.

How it works

Three mode options:

• scene_objects - captions the objects in the scene (from the workspace/panorama).
• trajectory_caption - captions what each trajectory shows, using the trajectory_set.
• prompt_refine - takes your prompt and improves it.

The model_id list is the whole Qwen3-VL lineup - 2B/4B/8B, Instruct or Thinking, each with an FP8 variant. quantization offers None (FP16), 8-bit, and the default 4-bit "VRAM-friendly", and attention_mode offers auto / sage / flash_attention_2 / sdpa. The 4-bit default is the right call: these VLMs are not small, and you're probably running one alongside a diffusion model.

Generation controls (max_new_tokens, temperature 0.6, top_p 0.9, num_beams, repetition_penalty 1.2, seed) behave like any HF generation call. max_image_edge (768) downscales input images so the VLM isn't choked with pixels it can't use. keep_model_loaded (true) keeps the weights resident - turn it off if you need the VRAM between runs. write_results saves the captions into the workspace so the rest of the pipeline can reuse them.

The inputs that matter

• workspace - required, grounds the whole thing in a scene.
• mode - pick what job you're doing.
• model_id + quantization - 4B-Instruct at 4-bit is the sweet spot for most setups.

Outputs

• llm_context (HYWORLD2_LLM_CONTEXT) - consumed by expansion nodes that take captions.
• text - the generated caption/refinement, for you to read and sanity-check.

Install & troubleshooting

Pack install: ComfyUI Manager (search HY-World 2.0) or clone + pip install -r requirements.txt + python install.py. This node runs the models through Hugging Face transformers (in requirements), and 4-bit loading needs bitsandbytes - also in requirements, but on some CUDA versions it's a fiddly install, so expect the occasional first-run hiccup there. FP8 variants additionally want a GPU with FP8 support.

First run downloads the model; that's a multi-GB pull, not a bug. If inference is glacial, check that you didn't land on an FP16 8B model on a 16 GB card - drop to 4-bit, or the 2B, and let keep_model_loaded do its job between runs. And if captions look wrong, that's not the node failing - it's the model being a model. prompt_refine mode is a nice way to see what it's actually paying attention to.