LiteLLMImageCaptioningProvider

Captioning is the unglamorous workhorse of every serious ComfyUI user: you generate a batch of images and you want a text description of each one, for training data, for tagging, or just to sort through results. LiteLLMImageCaptioningProvider turns a vision-capable model into a ready-made "describe this image" function that fits the pack's callable pattern - wire it once, and any node that accepts a CALLABLE can caption images.

Like LiteLLMCompletionProvider, this node doesn't answer anything itself. It outputs a Captioning function - a callable that takes an image (and optionally a prompt) and returns the caption text. That's the design: you configure the model and default prompt here, then pass the function around to wherever captions are needed.

What it does

At build time it takes your optional image input and encodes it to a base64 JPEG, embedding it into the message as a vision image_url alongside the prompt (default "Describe the image:"). The resulting callable can be invoked with a new image and/or a new prompt on the fly - so you can build the provider once with a template prompt and then feed it different images per call.

The image handling is ComfyUI-native: it converts the IMAGE tensor (BHWC, floats in [0,1]) into a JPEG at quality 100, base64-encodes it, and ships it to the model in the multimodal message format. No file round-trip, no manual base64.

The inputs

• model - a LITELLM_MODEL. Default is anthropic/claude-3.5-sonnet - note that's a vision model, which is exactly what you want here.
• max_tokens (250), temperature / top_p (0.5), frequency_penalty / presence_penalty (0) - sampling dials.
• image (optional) - the seed image; captions for subsequent calls come from new_image on the returned function.
• prompt (optional, multiline) - the captioning instruction.
• messages (optional) - preloaded conversation.

Output: Captioning function (CALLABLE).

Installing

It's part of ComfyUI_LiteLLM. ComfyUI Manager, search "ComfyUI_LiteLLM", or:

cd ComfyUI/custom_nodes
git clone https://github.com/Hopping-Mad-Games/ComfyUI_LiteLLM
cd ComfyUI_LiteLLM
pip install -r requirements.txt

Restart, provider key set, find it under ETK/LLM/LiteLLM.

Where people get burned

The vision-model requirement is the one that bites. Text-only models reject image content - connect an image and pick gpt-3.5-turbo and you'll get an error or a nonsense answer. Use a known-vision model (Claude 3.5+, gpt-4o, Gemini variants). Second, the default prompt is a bare "Describe the image:" - that's a low-effort caption that will get you low-effort captions. If you're building training data, write a real captioning instruction ("Describe the subject, pose, lighting, and style in detail...") the way you'd prompt any model, because the model will follow it. Third, watch the token bill: images at JPEG quality 100 are fat messages, and captioning a large batch is a large bill. If you're captioning hundreds of images, size down before encoding - the provider has no resize step, so what you feed it is what it sends. And remember, the output is a function, not text - it only does visible work when something downstream actually calls it.