Run Phi Multimodal

Run Phi Multimodal is the generation node paired with Load Phi Multimodal, and it's the one with the widest range in the whole pack: give it just an instruction and it's a text-only chat node; hand it an image tensor and it describes, tags, or analyzes what you gave it. That flexibility is the point of Phi-4-multimodal - one model that doesn't care whether you're feeding it words, pixels, or both.

For a ComfyUI workflow this is a captioner on the newer, better model. Where the vision node is the lightweight option for a tight VRAM budget, this one trades heavier memory for smarter output: better instruction-following, less rambling descriptions, and the ability to run as a plain LLM when there's no image attached.

How it works

The node builds the chat prompt by hand using Phi's raw chat tokens - <|user|>...<|end|> and <|assistant|> - rather than a chat template call. If you connect an image, it converts the tensor to PIL, injects <|image_1|>, <|image_2|> placeholders before your instruction, and passes the actual images to the processor. Then it calls model.generate with your max_new_tokens and the generation config it received from the loader - the model's own recommended settings - plus num_logits_to_keep=1, a memory trick that skips computing logits for all but the final token.

Two niceties that differentiate it from the other run nodes:

• The prompt is a real output. You get the exact formatted prompt string back, so you can see precisely what the model received - no black box.
• Text-only is first-class. Leave the image input disconnected and the node still runs a clean chat completion. Useful when you want the smart Phi-4 for prompt rewriting and the vision side is just along for the ride.

Inputs and outputs

Required:

• phi_model / phi_processor / phi_config - the three outputs of Load Phi Multimodal, all three needed.
• instruction - what to ask. Default "Describe this image".
• max_new_tokens - default 1000, the highest cap in the pack, matching a model that likes to talk.

Optional:

• image - a ComfyUI IMAGE tensor. Leave it empty for text-only.

Outputs - two strings:

• prompt - the full formatted prompt (chat tokens and image placeholders included). Great for debugging; also readable proof of what happened.
• response - the generated text. Wire it into Preview Text or anywhere a prompt string goes.

Install and the things that bite

Same pack install as everything else here (Manager → "Phi", or clone + pip install -r requirements.txt), and the model needs to be in place at ComfyUI/models/microsoft/Phi-4-multimodal-instruct - downloaded manually, because this pack never auto-downloads.

Real-world friction:

• Your GPU has to run flash attention. The loader this node expects hard-codes flash_attention_2, so everything that limits the loader limits you: Ampere-or-newer GPU, flash_attn installed (prebuilt wheel on Windows, per the README). If the load node works, this one generally does too - the pain is upstream.
• Memory is the real cost. 5.6B params in fp16 plus image crops. On a 16GB card alongside a diffusion checkpoint it's a juggling act; ComfyUI may offload, but generation will crawl when it does.
• No temperature/sampling knobs here. Unlike the vision run node, this one exposes only instruction and token cap; the generation config from the model drives the rest. If you want to fiddle with sampling, you're in the wrong node - that's a Run Phi Vision or Run Phi conversation.
• Long default cap. 1000 tokens is a lot of generated text; if captions feel endless, drop max_new_tokens rather than fighting the model in the prompt.