KY Caption Image by openai-protocol local LLM services

This is the vision-input sibling of this pack's KY_OpenAIChat - same underlying pattern, a generic client for anything speaking the OpenAI-style chat completions API or Ollama's protocol, but this time actually wired to send an image, not just text. Point it at a local vision-language model server, hand it one image, and it sends the image plus a prompt and gets a caption back.

The tell in the defaults, again

Same fingerprint as KY_OpenAIChat: base_url defaults to http://127.0.0.1:23333 (LMDeploy's default port), api_key defaults to the placeholder sk-0123456. But system_prompt here defaults to something different and specific to this node - "You are a movie scene director" - a distinct personal preset from KY_OpenAIChat's translator default, and another small, genuine window into what the author actually uses this node for day to day. caption_prompt defaults to "Describe this image without any speculations," a reasonably careful instruction that pushes the model toward describing only what's actually visible.

The model/ollama_model dropdowns list real, current vision-language models: gpt-4o/gpt-4o-mini for hosted use, and for local serving, names like qwen2.5vl:7b/qwen2.5vl:32b, OpenGVLab/InternVL3-8B, and an Ollama-packaged JoyCaption variant - JoyCaption specifically has an established community track record as a captioning model people actually run locally for exactly this kind of natural-language image description work, not a name invented for this node's dropdown.

Inputs that matter

• image_in - the required IMAGE input, the one thing that makes this node different from KY_OpenAIChat.
• protocol - openai or ollama, picking the API shape.
• model/ollama_model/custom_model - pick from the dropdown, or type any model name your own server actually has loaded via custom_model.
• system_prompt/caption_prompt - the persona and the instruction; both worth actually rewriting for your use case rather than leaving at the "movie director" default unless that's genuinely the tone you want.
• response_format - text (default) or json_object. Worth using json_object if you plan to feed the caption into something structured downstream, like this pack's own KY_MergeToJSON.
• Standard sampling knobs: max_tokens, temperature, top_p, frequency_penalty, presence_penalty.
• base_url/api_key - swap to a real hosted endpoint and key if you're not running a local server.

Single output: text_out, the caption as a string.

Installing it

ComfyUI Manager: search ComfyUI-KYNode, install, restart. Or git clone https://github.com/yorkane/ComfyUI-KYNode into custom_nodes and restart. No model weights ship with the node itself - you need a running OpenAI-protocol or Ollama server with a vision-capable model actually loaded; this is a thin HTTP client, not an inference engine.

Common issues

Out of the box, 127.0.0.1:23333 isn't listening on a fresh install - you need a real OpenAI-protocol server running there (LMDeploy, vLLM, LM Studio) or ollama serve with a vision model already pulled, or repoint base_url/api_key at a genuine hosted API.

If you switch protocol to ollama, remember the ollama_model dropdown only lists names - it doesn't install anything. Run ollama pull <model> for whichever one you pick before the node will find it.

Not every model behind a generic OpenAI-protocol endpoint actually supports vision input even if the server itself is running fine - a text-only model loaded behind an OpenAI-compatible API will either error or silently ignore the image and caption based on nothing, depending on how that specific server handles it. Confirm the model you've picked is genuinely a vision-language model, not just any chat model.

Sampling parameters that seem to have no effect are usually a backend limitation, not a bug here - some minimal local inference servers implement only a subset of the OpenAI sampling API and quietly ignore the rest.