CXH_Florence2Run

CXH_Florence2Run is the do-everything inference node for the pack's Florence-2 side. Give it a model from CXH_DownloadAndLoadFlorence2Model and an image, pick a task, and it'll caption, tag, OCR, find regions, or answer questions about the image - and hand you the annotated image, a mask, the text, and raw JSON. It's the node you reach for when you want fast, local, one-shot vision without loading an 8B LLM.

The README's speed ranking (florence2 fastest, then MiniCPM 2.6, then JoyCaption slowest) is worth internalizing: Florence 2 is the cheap-and-cheerful option. For plain dataset captioning it's the community's standard complement to JoyCaption - many people use Florence 2 for a first pass and save JoyCaption for quality sets.

How it works

The task dropdown maps to Florence 2's built-in prompt tokens:

• caption / detailed_caption / more_detailed_caption - three tiers of image description. The default is more_detailed_caption.
• region_caption, dense_region_caption, region_proposal - describe or locate objects, drawing labeled bounding boxes onto the image.
• caption_to_phrase_grounding, referring_expression_segmentation - match a phrase to regions/polygons, producing a segmentation mask.
• ocr, ocr_with_region - extract text, with or without bounding boxes.
• docvqa - answer questions about a document (requires text_input; it raises an error if you leave it empty).
• mixed_caption(PromptGen 1.5) and generate_tags(PromptGen 1.5) - the extra tasks from the default MiaoshouAI PromptGen model, blending captions with SD-style tags.

The model generates, the processor decodes, and for the region/OCR tasks it draws boxes with matplotlib, returns a MASK of the selected regions, and puts the structured output in JSON. Region tasks draw an annotated IMAGE; caption/OCR tasks pass the original image through.

Inputs and outputs

Inputs: image, florence2_model, task, text_input (multiline - used by docvqa, phrase grounding, and OCR/region tasks), fill_mask (whether detected regions fill a mask), keep_model_loaded (offload after run when false - keep it off unless you're looping), max_new_tokens, num_beams, do_sample, output_mask_select (comma-separated region indexes to isolate in the mask), seed.

Outputs, all four: image (annotated IMAGE), mask (MASK), caption (STRING), data (JSON with boxes/labels).

Installing

Same pack install - ComfyUI Manager (search "Comfyui_CXH_joy_caption") or:

cd ComfyUI/custom_nodes
git clone https://github.com/StartHua/Comfyui_CXH_joy_caption
cd Comfyui_CXH_joy_caption
python -m pip install -r requirements.txt

Troubleshooting

• docvqa errors with empty text: that's by design - text_input is required for that task.
• flash_attn missing: the loader already patches around it, so you shouldn't hit this. If you do, you're on a model file that the workaround missed.
• Slow on base Florence-2-large: normal. Use keep_model_loaded=True when looping over a batch; leave it off for one-offs.
• Annotated-image text is tiny on high-res images - a matplotlib label-size quirk, not a failure. OCR region labels behave similarly.