Florence-2 Multitask Vision

This is the node that actually runs Florence-2, and it does more than you probably think. The dropdown has fourteen tasks - captioning, object detection, region captioning, phrase grounding, segmentation, OCR - so one node replaces a whole shelf of single-purpose tools. In practice, though, almost everyone reaches for it for one job: turning images into caption .txt files for LoRA training. That's the workflow that made this pack famous on Civitai, and it's still the right reason to use it. Florence-2 is the standard "good enough" captioner for Flux-style datasets (the KB's training essays put it alongside JoyCaption and way ahead of BLIP), and a detailed caption run piped to a text saver gives you training captions in minutes.

The mechanism is straightforward if you know the model. Each friendly task name maps to a Florence-2 task token - caption becomes <CAPTION>, object detection becomes <OD>, and so on. The processor builds the prompt, model.generate() runs with your token/beam settings, and the result gets parsed back into structured data. The text_input field is appended to the prompt, but only the six grounding/segmentation/region tasks actually use it: caption-to-phrase-grounding, referring-expression segmentation, region-to-segmentation, open-vocabulary detection, region-to-category, and region-to-description. For plain captioning, leave it empty.

The inputs that matter, in order:

• task - pick your poison. detailed caption or more detailed caption for training data, object detection when you want boxes.
• text_input - only for the six region/grounding tasks above. For "caption to phrase grounding" you type the phrase you want located.
• max_new_tokens (1024) and num_beams (3) - leave them unless a caption is getting cut off, then bump tokens.
• do_sample - keep false for deterministic captions, true for variety.
• fill_mask - only affects the segmentation previews: it fills detected regions with solid color instead of outlines.

The three outputs are where this node earns its keep:

• preview (IMAGE) - the input image with boxes/polygons/OCR boxes drawn on. For plain caption tasks it's just the input image unchanged, which surprises people; that's expected, there's nothing to draw.
• string (STRING) - the parsed result as text. For captioning this is your caption; for detection it's a Python-dict-as-string you can inspect or save. Wire this into a text-save node for LoRA datasets.
• F_BBOXES - the structured detection/segmentation data (boxes plus labels, or polygons plus labels, plus image width/height). This feeds the pack's two postprocess nodes.

One honest warning before you build a big batch pipeline: this node only processes image[0] - the first image in a batch, silently. A folder-of-images captioning setup will caption one image and call it a day. That's the single biggest reason people migrate to Kijai's fork (ComfyUI-Florence2), which handles batches properly, has more nodes, and is what the README now recommends. Since the models share the models/LLM/ folder, the switch costs you a workflow rewrite but no re-downloads.

Install as usual via ComfyUI Manager (search "ComfyUI-Florence-2") or:

cd ComfyUI/custom_nodes
git clone https://github.com/spacepxl/ComfyUI-Florence-2

The pack ships no requirements.txt, so make sure transformers and matplotlib are installed (pip install transformers matplotlib) or you'll get a ModuleNotFoundError at load time. Detection and segmentation previews go through matplotlib, which is why a plain caption doesn't need it but object detection will.

If you're here to caption a training set: load the model, set task to detailed caption, read the string output, done. If you're here for masks and boxes, read the postprocess node docs - that's where the coordinates turn into something usable.