Ollama Vision Dual Planner

The Ollama Vision Dual Planner is the pack's most flexible planner, and the "Dual" is the whole point: it separates the two questions a vision-model planner is really answering into two separate prompt boxes. First, what is this image - you control exactly how the vision model describes it. Second, what do you want to do with it - your generation intent. The Style Planner fuses those into one step; this node lets you pull them apart.

That split is genuinely useful once you've fought the Style Planner's one-shot prompt. Say you want to ignore the background and match only the lighting. In the Dual Planner you set vision_prompt to "Describe only the lighting and color grading of this image" and keep your generation_prompt as whatever you're actually trying to make. The vision model describes what you asked it to, and the generation plan is built from that description plus your intent - not from whatever the model happened to notice on its own.

How it works

Mechanically it's the same engine as the Style Planner: your image is encoded to a base64 PNG and sent to Ollama's /api/chat with the vision_prompt as the analysis question, the generation_prompt as the generation instruction, and the compacted model_registry.json so the model has an actual menu of checkpoints and LoRAs to choose from. The same denoise playbook applies - low for targeted edits, high for style transfer, 1.0 for text2img - and the same fallback kicks in if the vision call fails (keyword heuristics, planning blind).

One difference: its task_hint menu is narrower (auto, img2img, text2img), because the Dual Planner is designed around image-guided generation rather than architecture picking. And you'll notice the default vision_prompt is a solid general-purpose ask: "Describe the style, composition, lighting, and key elements of this image." That default works fine - the node earns its name the moment you replace it.

The inputs that matter

• image - the reference (IMAGE tensor from LoadImage).
• vision_prompt - the analysis question for the vision model. This is the one to customize.
• generation_prompt - your actual intent, e.g. "make it cyberpunk, neon, high contrast."
• ollama_model - default llava:7b; needs to be vision-capable.
• max_vram, aspect_ratio, base_size - the usual budgeting and resolution controls.

The outputs that matter

The familiar 24-output plan: checkpoint, loras, lora_strengths, model_type, denoise, positive_prompt, negative_prompt, steps, cfg, sampler_name, scheduler, width, height, seed, vae_name/clip_name, plus plan_json. Same wiring as the other planners - into DynamicCheckpointLoader, DynamicLoraStack, and the sampler. There's even a workflow in the repo (workflows/ollama_dual_prompt_workflow.json) that shows the full chain, image in one end, planned img2img generation out the other.

How to install it

Part of GeekatplayStudio/ComfyUI-cluster. ComfyUI Manager (search "ComfyUI-cluster") or:

cd ComfyUI/custom_nodes
git clone https://github.com/GeekatplayStudio/ComfyUI-cluster

Restart ComfyUI, and make sure a vision model is pulled:

ollama pull llava:7b

No Python dependencies for the node itself.

Troubleshooting

Same silent-fallback trap as the Style Planner: no vision model, or a non-vision ollama_model, and the node quietly plans on heuristics without ever describing your image. Check the console when plans feel disconnected from the reference. And if your generations are ignoring the image's content but keeping its structure, look at your vision_prompt - if it asked only about lighting, don't expect the model to also have captured the composition. That's not a bug, it's the feature.