Bernini MLLM Prompt Enhancer (GGUF)

Bernini is two models wearing a trench coat: a 7B multimodal LLM that plans the edit, and a 14B diffusion transformer that draws it. The renderer has worked in ComfyUI since the day ByteDance dropped the weights - Kijai's integration is how nearly everyone runs it. The planner is the other half, the "latent semantic planning" in the paper title, and it famously never got a working ComfyUI path. So the whole community hand-writes those long, indexed, instruction-style prompts ("the man from image0, wearing the shirt from image2, in the beach sunset from image4…") and hopes for the best. This node is the gap-filler: it runs the Bernini-MLLM planner locally on a GGUF Qwen2.5-VL 7B. No API, no key, no cloud - the name "Prompt Enhancer" undersells what it is.

How it works

The trick that makes it interesting is the subprocess. ComfyUI loads your graph, but the actual model lives in a separate Python process the node spawns, talks to over JSON lines on stdin/stdout, and kills after every run. That's the "zero VRAM residue" claim - after the node finishes, the llama.cpp process is gone and your VRAM is fully back, which matters when the renderer behind it wants every megabyte. The catch: the model reloads from disk on every execution, so the first run each session takes a slow ~20 seconds. That's the honest trade for not holding VRAM hostage.

When it runs, it samples frames from your video (uniform by default, or smart_frames - a frame-diff sampler that keeps frame 0 and fills the rest with the highest-change frames, which gives a vision model more to chew on than 5 evenly spaced frames would), base64-encodes them, and builds a multimodal chat message from a per-task system prompt. The output gets split on a FINAL_PROMPT marker into the finished prompt and the reasoning trace.

The inputs that actually matter

• model + mmproj - dropdowns auto-scanned from ComfyUI/models/clip/. The README points at mradermacher's Bernini-MLLM-Qwen2.5-VL-7B-GGUF, Q4_K_M recommended. Get the matching mmproj or the model can't see anything.
• task_type - 13 of them: t2v, t2i, v2v (edit source video), i2v, r2v (reference-to-video), rv2v, fl2v (describe the transition between a start and end frame), ads2v, and more. Defaults to v2v.
• variant - narrows the job with a style: cinematic (t2v), anime (t2v), motion (r2v), 3dreal (rv2v), and a few more. Leave at none until you know you want a flavor.
• template_mode - structured gives you the five-section RULES trace; official is a single pass that mirrors the paper. Structured is more useful for debugging what the model "saw."
• smart_frames + video_frames - flip smart frames on for videos with lots of motion.
• prompt - optional! Leave it empty and the node uses a default instruction per task type. This is surprisingly handy for lazy t2v.
• The usual suspects - temperature, seed, max_tokens, n_ctx, n_gpu_layers (-1 = all GPU).

On the input side you can feed source_video, reference_video, and up to three reference_image_0/1/2 - which covers Bernini's whole reference-driven menu.

Outputs

• enhanced_prompt (STRING) - wire this into your Bernini renderer's prompt input.
• structured_plan (STRING) - the "RULES" reasoning trace; slap it on a text viewer to see what the planner decided before it wrote the prompt.
• source_video, reference_video, reference_image_0/1/2 - straight passthroughs so you can chain nodes without re-loading assets.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/djdzzzz/ComfyUI-Bernini-PromptEnhancer.git
cd ComfyUI-Bernini-PromptEnhancer
pip install -r requirements.txt

Or grab it from ComfyUI Manager - it's on the registry as ComfyUI-Bernini-PromptEnhancer. Then download the GGUF model and its mmproj into ComfyUI/models/clip/ and restart ComfyUI - the dropdowns are scanned at startup, so a model you just dropped in won't appear until you do. requirements.txt is just llama-cpp-python, torch, numpy, Pillow, so the only heavy lift is llama.cpp compiling (it usually ships prebuilt wheels).

Where people get burned

• Model not in the dropdown after download. Restart. The scan happens at boot.
• Nothing comes back / "worker error" in the console. The node retries once with a fresh worker, then gives up - the traceback lands in your ComfyUI console, so read it. A missing or mismatched mmproj is the usual cause.
• OOM. Drop n_gpu_layers (from -1 to ~20), n_ctx (8192→4096), image_max_side (512→384), or video_frames (3→2). The INSTALL doc lays out the same ladder.
• Expecting magic. A local enhancer removes the blank-page problem; it doesn't out-write a great hand-crafted prompt. Use it to get 80% of the way there, then edit.

The pack is brand-new - djdzzzz published it with zero community track record yet - so treat it as promising rather than battle-tested, and check the repo for updates before trusting a workflow to it.