📝 Florence PromptGen

The blank-prompt problem, solved by a 0.23B model

You have an image you like. You want more of it, or to move it into a video workflow, or to caption 400 files for a LoRA set - and the blocker is always the same empty text box. Florence PromptGen fills the box in: hand it an IMAGE, get back an English prompt as a STRING.

It wraps MiaoshouAI's Florence-2 PromptGen v2.0, a fine-tune of Microsoft's Florence-2 - the 0.23B/0.77B captioner that's been the community's lightweight default since June 2024. Small, MIT-licensed, and it doesn't moralise about what it's looking at, which is why people kept it for caption work instead of paying an API. PromptGen v2.0 sharpens it toward one job: write something you can paste into a sampler.

So: seed img2img from a photo, generate the prompt for an image-to-video graph, or batch a folder into caption text. No key, no account, no cloud bill.

How it works

Florence-2 is a unified prompt-driven model. You feed it the image plus a task token, and the same weights do captioning, OCR, detection or region analysis depending on which token you sent. That's the entire trick behind the mode dropdown - each option is a different task token, no second checkpoint involved.

Implementation detail worth knowing: this node doesn't run the original Florence-2 code. It loads the checkpoint with native transformers Florence2 classes (transformers>=4.57.6 is a hard requirement), remaps the PromptGen weights into the modern layout on load, and registers a custom attention path that routes through ComfyUI's own optimized attention. Practically that means no Flash Attention install, no remote code execution from the model repo, no separate Florence node pack to keep alive.

The inputs that matter

You'll set maybe three.

• mode - the task token. Detailed prompt is the default and the right first choice. Short caption for one-liners, Tags for tag-ish output, Caption + tags + composition for the kitchen sink. OCR is the odd one out: it transcribes visible text in its original language and does nothing else.
• model - base or large. Base is fast and fits anywhere an SDXL checkpoint fits. Large writes better sentences at several times the memory; try it only if base keeps missing things.
• max_new_tokens / num_beams - speed knobs, defaulting to 256 tokens and 1 beam for throughput. Three beams buys noticeably better phrasing at roughly triple the time. Good trade for one hero image, bad one for 500.
• keep_model_loaded - off by default, and the node unloads the model when it finishes. Captioning a whole dataset in one session? Tick it.
• image - RGB or RGBA, single or batch. Alpha is ignored; only colour channels are read.

One output: prompt, a STRING. Note that it's a list - one string per image in the batch. Feed it to CLIP Text Encode and ComfyUI iterates: six input images means your downstream chain runs six times, once per prompt, not one six-prompt batch. Usually what you want for captioning, rarely what you want otherwise.

Install

Manager: search the pack name SDVN_Comfy_node. Or by hand:

cd ComfyUI/custom_nodes
git clone https://github.com/StableDiffusionVN/SDVN_Comfy_node
cd ../..
pip install -r custom_nodes/SDVN_Comfy_node/requirements.txt

Restart. No model to hunt down - first execution pulls the PromptGen weights anonymously into ComfyUI/models/LLM/Florence-2-base-PromptGen-v2.0 (or the large folder) and reuses them after that. No HF token, no gated repo.

Be aware this is a big pack: openai, google-genai, ultralytics, segment_anything, transformers>=4.57.6. You're installing all of it for one captioner, and the pack's __init__.py re-checks those versions on every startup and pip-installs anything out of spec. Convenient - until it upgrades a package another node pack pinned.

Where people get burned

That transformers>=4.57.6 floor is the usual culprit. If another pack in your environment holds transformers lower, this node either won't import or triggers the startup checker into a pip install that shifts the version under your other packs. After any such update, restart and read the first 20 console lines.

First run looks frozen. It's downloading a few hundred MB before the progress bar moves. Watch the console, not the queue.

A caption is not a reproduction. Florence describes the image accurately enough that you believe it, and the prompt still often won't regenerate that image. Every VLM captioner shares this: faithful in language, loose in specifics. For an anime base trained on booru tags, the community answer is to run Florence and a WD14 tagger and concatenate - Florence gives scene context, WD14 gives the atomic tags Florence never emits.

Multi-subject images get scrambled. Two people in frame and Florence will swap who's wearing what. No parameter fixes that; caption those by hand.

OCR on an image with no text returns almost nothing, which reads as a bug and isn't.

Wire prompt into CLIP Text Encode for variants, into a text-save node for datasets, or into an LLM enhancer to have it rewritten for a specific model's taste. A good starting point, a mediocre final prompt - the honest trade for zero API calls.