Load Phi Multimodal

Load Phi Multimodal is the newest loader in the ComfyUI-Phi pack, added in v4.0.0, and it's the one for Phi-4-multimodal-instruct - Microsoft's 5.6B-parameter model that genuinely does image, audio, and text. The selling point over the vision loader is simple: Phi-4 is a later, smarter generation, and the model is multimodal in a deeper way - it can talk about images, yes, but it's a general chat model too. Same graph, same wiring, bigger brain.

One honest warning up front: this loader is the most demanding node in the pack, and not just in VRAM. If you're on a pre-Ampere GPU (GTX 10-series, 20-series), stop and read the gotchas below before you build around it.

How it works

Same family recipe as the other loaders - resolve ComfyUI/models/microsoft/Phi-4-multimodal-instruct, then from_pretrained with local_files_only=True, device_map="cuda", torch_dtype="auto", trust_remote_code=True - with three differences:

1. It loads an AutoProcessor instead of just a tokenizer, because it has to turn images (and audio) into model inputs.
2. It grabs a GenerationConfig from the model folder and hands it out as a third output, so Run Phi Multimodal can generate using the model's own recommended defaults.
3. The model is loaded with _attn_implementation='flash_attention_2' hard-coded. Unlike Load Phi Vision, there's no attention dropdown. The source even has the comment - "if you do not use Ampere or later GPUs, change attention to eager" - but there's no widget to change it. On unsupported hardware, this loader fails at load time unless you're willing to edit the Python file yourself.

Inputs and outputs

Just one input:

• model - dropdown with the single choice Phi-4-multimodal-instruct.

Three outputs:

• phi_model - the loaded model.
• phi_processor - the AutoProcessor (tokenizer + image/audio preprocessor).
• phi_config - the model's GenerationConfig, passed through to the run node.

All three feed Run Phi Multimodal; nothing else accepts these types.

Installing it

Pack install is the standard routine (Manager → search "Phi", or git clone https://github.com/alexisrolland/ComfyUI-Phi into custom_nodes and install requirements.txt). The model is the big one:

cd ComfyUI/models/microsoft
git clone https://huggingface.co/microsoft/Phi-4-multimodal-instruct

That repo is roughly 11GB+ of safetensors; the folder name must stay exactly Phi-4-multimodal-instruct. As with every loader here, local_files_only=True means no download happens on its own.

The gotchas that actually matter

• Flash attention is mandatory, not optional. Hard-coded in the source. You need (a) an Ampere-or-newer NVIDIA GPU, and (b) the flash_attn library installed in ComfyUI's Python. On Windows the README points at a prebuilt flash_attn wheel for Python 3.12 plus triton-windows. If loading errors immediately, this is the first thing to check - not the model download.
• It's the heaviest model in the pack. 5.6B params in fp16 is ~11GB of weights before a single image is processed. Realistically you're looking at a 16GB+ card if you also want a diffusion checkpoint loaded. This is a "run it, save the workflow, use it for captioning, and forget it's sharing your GPU" kind of node.
• No dtype/attention escape hatch. The vision loader gives you both widgets; this one gives you neither. If you hit the flash-attention wall on older hardware, your options are editing the node source or using the vision path instead.