GR Florence 2 Caption Generator

Most Florence-2 nodes give you one job: point it at an image, get a caption back. This one gives you fourteen different model checkpoints, fifteen task modes, and a built-in image-style classifier, then hands the result off to an ollama output like it expects you to keep going. That's the tell - this node isn't built for one-off captions, it's built as the front end of a prompt-generation pipeline, probably feeding a local LLM node that rewrites or expands whatever Florence-2 pulled out of the picture.

Florence-2 itself is a solid pick for that job. Microsoft's model is well-regarded in this community specifically for natural-language captions - it's the go-to alongside JoyCaption when you're writing prompts for Flux-style models rather than tagging for Illustrious/Pony, and it plays nicely paired with other tools in a pipeline rather than working alone. This node leans into that: instead of the stock microsoft/Florence-2-large weights, its default is MiaoshouAI/Florence-2-large-PromptGen-v2.0, a finetune specifically trained to produce prompt-shaped text rather than plain photo descriptions.

How it works

You pick a model (14 choices - the PromptGen finetunes, vanilla Microsoft base/ft weights, a DocVQA specialist, CogFlorence, an SD3-tuned captioner) and a task (15 choices: caption, detailed_caption, more_detailed_caption, region and grounding tasks, ocr, docvqa, prompt_gen_tags, and a few more). Feed it an image, and it runs that task through the chosen checkpoint. Some tasks need more than an image - grounding and referring-expression tasks want a phrase to search for, which is what text_input is for; for plain captioning tasks you can leave it blank.

On top of the raw Florence-2 output, the node also classifies the image against an image_types list (58 style buckets - abstract, cyberpunk, documentary, and dozens more) and a media_type (image, image upscale, video, and a couple of "subtle video" variants). That's not something Florence-2 does natively; it's this node layering its own classification on top, which is why you get four separate string outputs instead of one caption.

The inputs and outputs that matter

• task and model - the two choices that actually decide what you get. Start with caption or detailed_caption on the default PromptGen model unless you specifically need OCR, region boxes, or a document Q&A answer.
• image (optional input) - yes, optional on the schema, but there's not much point running this node without one.
• max_new_tokens (default 1024) and num_beams (default 3, up to 64) - the usual generation knobs. Higher beams cost more time and VRAM for marginal quality gains; 64 is there if you want it, not because you should reach for it.

Four outputs come out the other side: prompt (the Florence-2 result), image_type_description, media_description, and ollama - wire that last one into a local LLM node if your workflow rewrites captions before they hit the sampler.

How to install it

Search GraftingRayman in ComfyUI Manager, or manually:

cd ComfyUI/custom_nodes
git clone https://github.com/GraftingRayman/ComfyUI_GraftingRayman

Then restart. The pack has one dependency the README is blunt about: you need OpenAI's CLIP installed separately, or the nodes fail to import entirely. For portable installs, run .\python_embeded\python.exe -m pip install git+https://github.com/openai/CLIP.git from your ComfyUI folder; on a system Python install it's pip install git+https://github.com/openai/CLIP.git. This isn't optional housekeeping - the README says explicitly that without it, import fails.

Common issues & troubleshooting

The whole node pack silently fails to load. Nine times out of ten this is the missing CLIP dependency above - check your console log at startup for an import error from ComfyUI_GraftingRayman and run the pip command before filing a bug.

First run is slow and downloads a lot. Whichever of the 14 model checkpoints you pick gets pulled from HuggingFace on first use - the PromptGen large variants are multi-gigabyte downloads. Budget for that on a cold node, and don't be surprised if switching models mid-session triggers another download.

Grounding/region tasks come back empty or wrong. Tasks like caption_to_phrase_grounding and referring_expression_segmentation need a real phrase in text_input to search for - leave it blank (the field's default) and you're asking the model to ground nothing.

Generation is taking forever. num_beams defaults to 3 but goes up to 64. That range exists for people chasing marginal quality on a batch job; for interactive use, keep it in single digits.