Load Phi Vision

Load Phi Vision is the loader for Phi-3.5-vision-instruct, the pack's first image-capable model. Its whole job is the same as Load Phi's - read weights off disk, give you a model and its processor - but the prize is different: a vision-language model that can describe the image currently sitting in your ComfyUI graph. That turns into captioning pipelines, automatic tagging, or "tell me what's wrong with this render" loops, all without leaving ComfyUI or hitting an API.

The reason you'd pick this over Load Phi is purely the vision half. The text-only Phi-3.5-mini can't look at anything; this one can. It's also the older of the pack's two vision loaders - the newer Load Phi Multimodal drives Phi-4, which is a bigger, smarter model. For a captioning sidecar on a modest card, this one is the lighter lift.

How it works

Mechanically it's the same pattern as the other loaders: resolve the ComfyUI/models/microsoft/Phi-3.5-vision-instruct folder, then AutoModelForCausalLM.from_pretrained(..., local_files_only=True, device_map="cuda", trust_remote_code=True). Two things differ. First, the model loads with _attn_implementation driven by an attention widget you can actually set - not hardcoded like the multimodal loader. Second, the processor is created with num_crops=16, which the model's own docs recommend for single-frame images (you'd use 4 for multi-frame/video). The vision model preprocesses the image by cropping it into tiles, so this number meaningfully changes both quality and VRAM.

Inputs and outputs

Required:

• model - one choice: Phi-3.5-vision-instruct.

Optional - and this is what makes this loader friendlier than the multimodal one:

• dtype - auto, float16, or bfloat16. Auto resolves to whatever the model wants, which is usually fp16. The author's tooltip flags bfloat16 as faster but less precise; if your card handles fp16 fine, leave it on auto.
• attention - eager (default) or flash_attention_2. Flash attention is faster on supported GPUs, but it requires the flash-attn library installed and an Ampere-or-newer GPU. Eager works everywhere; that's why it's the default here.

Outputs:

• phi_model - the loaded model.
• phi_processor - the AutoProcessor that turns an image tensor plus a prompt into model inputs.

Both feed Run Phi Vision. Like the rest of the pack, these are custom types - nothing else can consume them.

Installing it

Standard pack install (Manager → search "Phi", or clone into custom_nodes + pip install -r requirements.txt), then download the model by hand:

cd ComfyUI/models/microsoft
git clone https://huggingface.co/microsoft/Phi-3.5-vision-instruct

The folder must keep the exact name Phi-3.5-vision-instruct because the loader builds its path from it, and loading is local_files_only - no auto-download, missing folder means a red node.

Gotchas worth knowing

• VRAM is real. Phi-3.5-vision is 4.2B params, so ~8GB in fp16, and vision inference holds image crops on top of that. Budget for the model plus your diffusion pipeline sharing the card.
• The flash-attention option only helps if you have the wheel. On Windows the README walks you through installing a prebuilt flash_attn wheel for Python 3.12 and triton-windows. Skip that and stick with eager - it's the default for a reason.
• trust_remote_code. This model runs custom code from the repo. The README points at the forked Lexius/Phi-3.5-vision-instruct because it fixes issues in the original - if you get odd generation results from the stock microsoft repo, that fork is the known remedy.