VLMImageEditingPrompt

This node is the "brain" half of DreamOmni2 inside ComfyUI. You hand it an image and a rough instruction - "make the person in image 1 and the person in image 2 shake hands in space" - and it reads the images with a local vision-language model and writes back a long, detailed prompt that the rest of the pipeline actually samples from. It doesn't edit anything itself. It's the prompt writer, and in the pack's example workflows its output plugs straight into a CLIP Text Encode node.

DreamOmni2 (DVLab, late 2025) is a multimodal instruction-based editing-and-generation model that genuinely handles multi-image input on edit tasks where older Qwen-Image-Edit setups fizzle. This pack is a community reimplementation of that pipeline in ComfyUI, and VLMImageEditingPrompt is the VLM stage. Fair warning: this is a small, niche port (basically zero search footprint, Chinese-language README), not the mainstream RunningHub DreamOmni2 integration. It works - but you're an early adopter either way.

How it works

Under the hood the node loads a local Qwen2.5-VL-7B - the vlm-model folder from the DreamOmni2 HuggingFace repo - from ComfyUI/models/vlm_model, then quantizes it on load via bitsandbytes. The flow is textbook LLM-in-the-graph prompt enhancement: it builds a chat message containing your image(s), your instruction, and a suffix saying whether this is an edit or generation task, runs the chat template, and generates greedily (do_sample=False, so the same input gives the same output every time). It then strips a fixed wrapper the model emits and returns the inner text as a plain STRING.

The enhanced_prompt output is the whole point - wire it into a CLIP Text Encode node and it becomes the conditioning for the DreamOmni2 edit or generation LoRA on the Flux-family DiT that the included workflows use. Same pattern the community converged on everywhere: a VLM reads the image so the diffusion model doesn't have to guess what you meant.

The inputs that actually matter

Only a few of the six do anything for a beginner:

• image_1 (required) - the image being edited, or the content image in generate mode.
• instruction (required) - your rough instruction, plain English. Default is "Make the first image have the same style as the second image".
• task_type - edit or generate, defaults to edit. This just changes the task tag appended to your instruction; it should match the LoRA you load downstream.
• quantization - fp16 / int8 / int4, default int4. This is the real choice. A 7B VLM in fp16 is ~15GB, which is a hard sell when the diffusion model still needs the same card. int4 (NF4) is what the author ships as default for a reason; bump to fp16 only if you have VRAM to burn and want maximum fidelity.

The optional image_2 is your reference/style image for two-image prompts, and max_new_tokens (64–2048, default 1024) caps how long the VLM is allowed to write. 1024 is plenty - if the output reads like a novel, the model is rambling, not helping.

Installing it

ComfyUI Manager: search for ComfyUI-VLM-DreamOmni2. Or clone it yourself:

cd ComfyUI/custom_nodes
git clone https://github.com/swan7-py/ComfyUI-VLM-DreamOmni2
pip install -r ComfyUI-VLM-DreamOmni2/requirements.txt

Then restart ComfyUI. Dependencies are light and boring - torch, transformers, accelerate, bitsandbytes, pillow.

The model is the heavy part: download the vlm-model folder from xiabs/DreamOmni2 on HuggingFace - that's roughly 20GB in four shards. The node looks for it at ComfyUI/models/vlm_model, so rename the folder vlm-model → vlm_model (dash to underscore) or nothing loads. The edit and gen LoRAs in the same repo go in ComfyUI/models/loras/ only if you're building the full workflow; this node doesn't need them to run. If you have nunchaku installed, the example workflows will use it to accelerate the diffusion side.

Where people get burned

The trap in this node is silent. It wraps everything in a try/except, and when the model isn't loaded it doesn't error - it just returns your raw instruction (or instruction - reference style from second image) as the "enhanced" prompt. So the first symptom of a missing vlm_model folder isn't a red node, it's a workflow that "works" while outputting boring prompts that ignore your image. If the enhancement looks unchanged, check that folder and the console log first.

Second, the folder rename above is non-negotiable - skip it and you get the silent fallback, not a helpful message. Third, bitsandbytes is CUDA/ROCm-only, so on Apple Silicon the int4/int8 options won't fly; you're on fp16 (slow) there. And the [6:-7] wrapper slice that cleans the model output is fragile by construction - a prompt truncated at both ends means this node assumed the DreamOmni2 output format, not that you did something wrong. It's a reasonable trade for a 7B model doing a job that would otherwise be an API call, but treat this node as something to tinker with, not a finished product.