Ollama Vision Style Planner

The Ollama Vision Style Planner is the OllamaPromptPlanner's sibling with eyes. Same job - look at your prompt and the pack's registry, pick a checkpoint, LoRAs, sampler, resolution, the whole plan - but it also takes an image and actually sees it before deciding. A local vision model like llava:7b looks at your reference, identifies the style, and plans a generation setup matched to it. All local, no API key, no cloud.

This is the node for img2img work: you want "make this photo into a cyberpunk illustration" to result in a plan that picks a style-appropriate checkpoint, a sensible LoRA, and - crucially - a low denoise so the edit keeps the structure of your source instead of vaporizing it. That last bit is where the vision model earns its keep, because it's deciding what kind of edit you asked for.

How it works

It encodes your input image to a base64 PNG and sends it to Ollama's /api/chat with the image attached, along with your prompt, the compacted model registry, and a system prompt that instructs the model to analyze style and composition. The system prompt also hands it a denoise playbook: low (0.3–0.5) for minor edits like "fix eyes" or "change makeup," high (0.6–0.9) for style transfer like "make it anime," and 1.0 if the task is really text2img. That's the difference between a variation and a faithful edit, decided by a model that looked at the pixels.

Like the text planner, it has the same graceful-degradation design: if the vision call fails (Ollama down, no vision model pulled, bad response), it falls back to the text-only heuristic planner. Your workflow keeps running - but it's now planning blind, on keywords alone, which is worth knowing because the results will show it. Either way it then normalizes resolution from aspect_ratio/base_size, merges your user_negative, and blanks the negative prompt on Flux.

The inputs that matter

• image - the reference image (IMAGE tensor, e.g. from a LoadImage). First frame only if you feed a batch.
• prompt - what you want done with that image.
• ollama_model - default llava:7b; must be a vision-capable model or the vision path fails and you silently fall back.
• task_hint - auto, img2img, or text2img for the Style Planner (plus sdxl/sd15/flux).
• max_vram, aspect_ratio, base_size - same budgeting/resolution controls as the text planner.

The outputs that matter

The same 24-output plan as OllamaPromptPlanner: checkpoint, loras, lora_strengths, model_type, denoise (the one the vision model reasons about hardest), positive_prompt, negative_prompt, steps, cfg, sampler_name, scheduler, width, height, seed, vae_name/clip_name, and plan_json for debugging. Wire them into DynamicCheckpointLoader, DynamicLoraStack, and a KSampler with your VAEEncode in between for the img2img latent path.

How to install it

Same pack, same install as its siblings: ComfyUI Manager, search "ComfyUI-cluster", or

cd ComfyUI/custom_nodes
git clone https://github.com/GeekatplayStudio/ComfyUI-cluster

Restart ComfyUI. Then make sure you actually have a vision model:

ollama pull llava:7b

No Python dependencies for the nodes; Ollama is the only external piece.

Troubleshooting

The trap here is the silent fallback. If your ollama_model isn't a vision model, or isn't pulled, the node doesn't error - it quietly drops to keyword heuristics and plans without seeing the image. You'll notice in the output quality, not in an error message, so check the console. If plans look like they're ignoring your image entirely, that's the first thing to verify. Also set task_hint to img2img when you mean it; on auto the model may decide your prompt is text2img and run denoise 1.0, which destroys the reference.