Eric Qwen ControlNet Prompt Rewriter (Vision)

Here's the deal with ControlNet-guided generation: the ControlNet only gives you structure. Canny gives you edges, Depth gives you layout, Pose gives you a skeleton - but the model still needs a prompt that describes everything else, and a bare "cyberpunk warrior" is nowhere near enough detail to render a full scene. Eric Qwen ControlNet Prompt Rewriter closes that gap by sending your reference image and a one-line intent to a vision-language model, which writes the full descriptive prompt for you.

It's the ControlNet-specific sibling of the pack's Eric Qwen Prompt Rewriter, and the key difference is CN-awareness: the VLM is told which preprocessor is being used (soft contours vs skeleton vs depth map vs hard edges) so it knows what the structure already pins down and what it must describe in words. The docs' framing is exactly right: because the CN provides structure, not appearance, the prompt must describe everything the diffusion model needs to render.

How it works

The node talks to any OpenAI-compatible vision API - Ollama, LM Studio, DeepSeek, OpenAI - via a normal HTTP call. It sends the reference image (downscaled to image_max_side, default 1024) plus your creative_direction and cn_type, and the VLM returns a 200–400 word scene description. No model runs on your GPU; this is the API-wrapper pattern from the wider ComfyUI ecosystem.

Security note, straight from the KB: an API node that carries a credential and phones home is exactly the shape of the thing that got ComfyUI a malware scare once. This pack is on the up-and-up - keys come from environment variables or an api_keys.ini file that's git-ignored, never from the workflow JSON - but the pattern deserves your scrutiny on any node pack.

Inputs that matter

• image - the reference the VLM analyzes.
• creative_direction - your one-liner: "cyberpunk warrior in neon-lit alley", "same scene but underwater". The VLM expands it.
• cn_type - Softline, DWPose, Depth, CannyEdge, or PyraCanny. This is what makes the rewrite CN-aware.
• api_url / model - default http://localhost:1234/v1 (LM Studio) with qwen3-vl. Swap to Ollama's http://localhost:11434/v1 or OpenAI's endpoint; the model must accept images.
• passthrough - skip the VLM and return creative_direction untouched, for A/B testing the rewriter's value.
• temperature (0.7), max_tokens (2048), custom_instructions, language (English/Chinese), image_max_side - the quality and budget knobs.

Outputs: prompt (STRING) and cn_type_index (INT) - a 1–5 index (Softline=1 … PyraCanny=5) that can drive a downstream switch node to auto-select the matching preprocessor.

Wiring it

prompt goes into a generation node like UltraGen CN; cn_type_index can drive a node-switch that picks the preprocessor so the whole pipeline stays in sync.

Installing it

ComfyUI Manager → "Eric Qwen-Edit", or:

cd ComfyUI/custom_nodes
git clone https://github.com/EricRollei/Eric_Qwen_Edit_Experiments.git

Restart. The node itself needs no model - just a reachable vision API. For the full workflow you'll also want a Qwen-Image generation model (Qwen/Qwen-Image-2512) and optionally the InstantX ControlNet Union (~2.3 GB).

Troubleshooting

• Empty prompt or error - check the API is reachable and the model name is right for your server. LM Studio uses localhost:1234, Ollama localhost:11434.
• Model errors on the image - you picked a text-only model. The tooltip lists vision-capable ones (qwen3-vl, qwen2.5-vl:7b, llava, gpt-4o-mini).
• Keys not picked up - env vars beat the ini: DEEPSEEK_API_KEY, OPENAI_API_KEY, or the fallback ERIC_QWEN_API_KEY. Remember this pack stores keys outside workflows on purpose.