APNext Phi Custom Model Inference

PhiCustomModelInference (APNext Phi Custom Model Inference) is the loaded-up version of PhiModelInference. Same underlying model - Microsoft's Phi-3.5-vision - but with the extras this pack built for people captioning training data: dynamic LoRA-trigger substitution, multi-image blending, and a separate short caption tuned for the CLIP-L encoder. If you're generating captions for a LoRA dataset and want the subject token baked in automatically, this is the node you actually want.

Like the plain inference node it depends on a PhiModelLoader upstream to hand it a loaded pipeline. It just does more with it.

How it works

You feed it the phi_pipeline, your images, and a captioning instruction, and it runs Phi locally to describe them. The difference from the plain node is the templating layer on top.

The core inputs:

• phi_pipeline (PHI_MODEL_PIPELINE) - from PhiModelLoader. Required.
• images (IMAGE) - what to caption.
• custom_prompt - the base instruction; additive_prompt - extra text appended to it.
• generation_temperature (0.7) and max_output_tokens (2048) - same creativity and length controls as the plain node.
• fade_percentage (default 15) - blend strength when you pass multiple images.

The LoRA-captioning feature is the point of this node. Turn dynamic_prompt on and it substitutes placeholders in your instruction: ##TAG## becomes tag (default ohwx man, the classic Dreambooth trigger), ##SEX## becomes sex, ##PRONOUNS## becomes pronouns, and ##WORDS## becomes words (target length). So one template caption an entire dataset with the trigger word and correct pronouns already in place - which is exactly the tedious part of dataset prep you don't want to do by hand.

Three outputs: output (the full caption STRING), clip_l (a shorter caption aimed at a CLIP-L text encoder, handy for Flux-style dual-encoder training/inference), and faded_image (the blended composite of your inputs).

Installing it

ComfyUI Manager: search comfyui_dagthomas, install, restart. Or clone and pip install -r requirements.txt. Relies on transformers. No API key; Phi weights download through the loader on first use.

Common issues

Dynamic prompt did nothing. Two conditions have to both be true: dynamic_prompt is on, and your custom_prompt actually contains the ##TAG##-style placeholders. Overrode the prompt with plain text and no placeholders? Then there's nothing to substitute - the node just uses your text as-is.

"Missing phi_pipeline." You need a PhiModelLoader wired into this node; it can't load the model itself.

Wrong trigger in captions. The tag defaults to ohwx man. If your LoRA uses a different token (say sks woman), set tag, sex and pronouns to match, or every caption will describe the wrong subject.

Out of memory. Phi plus your image checkpoint on one GPU is tight. Caption first, then load the generation model, rather than holding both - and drop image_crops to 4 on the loader if you're still short.