Run Phi Vision

This is the payoff node of the vision side of the pack. Load Phi Vision puts Phi-3.5-vision-instruct in memory; Run Phi Vision takes a real ComfyUI image tensor, a text instruction, and produces a text response about what it sees. "Describe this image" is the default prompt for a reason - the thing is a captioner, and captioning is its killer use in a diffusion workflow. You can caption a batch of generated images for dataset prep, auto-tag outputs, or chain it into a "check my render for flaws" pass that feeds a fix-back into the graph.

The standout feature, added in v2.0.0, is few-shot prompting right in the node: you can hand it one example image plus the description you'd like, and it steers its output toward your format. That's the difference between getting a freeform paragraph and getting "subject, style, lighting, quality" tags every time.

How it works

Under the hood it's a manual generation loop rather than a pipeline. The node converts the input image tensor to PIL, builds <|image_1|>, <|image_2|> placeholders into your instruction, runs the chat template, feeds the whole thing through the processor with num_crops from the loader, and calls model.generate. Output tokens past the input are decoded and returned. The placeholders matter: Phi's vision training expects those markers, so the instruction output you get back literally contains them.

Inputs that matter

Required:

• phi_model / phi_processor - from Load Phi Vision.
• image - any ComfyUI IMAGE tensor; it'll accept a batch, though multi-image handling gets clunky.
• instruction - what to ask. Default "Describe this image". For tagging, tell it the format you want.
• do_sample - off by default. Turn it on and temperature (default 0.5) starts mattering; for captioning, deterministic is usually what you want.
• max_new_tokens - cap on the reply, default 500.

Optional:

• image_example + response_example - the few-shot pair. Provide both (a single image and its desired description text) and the node injects "here is an example of an image and its description" into the prompt. Provide only one and it's silently ignored - easy to miss.

Outputs - two strings:

• instruction - the full prompt that was sent, image placeholders included. Useful for debugging and for seeing exactly what the model got.
• response - the generated answer. This is the one you wire into Preview Text, or feed back as a prompt.

Install and troubleshooting

Pack install is the shared routine: ComfyUI Manager (search "Phi"), or clone alexisrolland/ComfyUI-Phi into custom_nodes and install its requirements.txt. Then make sure the vision model is downloaded:

cd ComfyUI/models/microsoft
git clone https://huggingface.co/microsoft/Phi-3.5-vision-instruct

No auto-download, exact folder name Phi-3.5-vision-instruct.

Where it bites:

• Few-shot silently doing nothing. If you wired in image_example but left response_example empty, the example is skipped entirely. Both-or-nothing.
• First run is slow and heavy. Vision generation holds the model (~8GB fp16) plus image crops. On a 12GB card alongside a checkpoint, expect pressure.
• The default answer can be rambly. This is a 4.2B model; it's not GPT-4. If captions are too chatty, push the instruction to demand "only a comma-separated list" and consider the few-shot pair to enforce your format.
• If generation errors on load, the usual cause is the stock microsoft repo misbehaving under trust_remote_code - the pack README recommends the Lexius/Phi-3.5-vision-instruct fork precisely because it fixes issues in the original.