Florence2Run

This is the node that actually does the work. You feed it an image, pick a task from a dropdown, and Florence-2 hands back a caption, a set of bounding boxes, a mask, or OCR text depending on what you asked for. One node, a dozen-plus jobs. If you've ever wanted your workflow to read an image instead of just render one, this is where that happens.

The single most common reason people reach for it: auto-captioning for LoRA training. Florence-2 is Microsoft's little vision-language model, and it's one of the two captioners the community actually trusts for natural-language datasets - the other being JoyCaption. The old standby BLIP is considered dead for this ("Blip is pure trash" is a direct quote from the training crowd), so if you're building a Flux, SD3, Z-Image or Qwen dataset and don't want to write hundreds of captions by hand, you point Florence2Run at each image and collect the text. A lot of people pair it with a WD14 tagger to get both a fluent sentence and a tag list.

But captioning is only one of its tricks. Florence-2 is a genuine multi-task model - same weights do object detection, region proposal, phrase grounding, referring-expression segmentation, plain OCR, OCR-with-regions, and (in this fork specifically) DocVQA, where you ask a question about a document image and get an answer back. That last one is kijai's addition to the standard node, handy for receipts, forms and scanned text.

How it works

Florence-2 is a sequence-to-sequence model trained on a giant annotation set (Microsoft's FLD-5B - billions of labels across 126M images). Internally, each "task" is just a special prompt token the model was taught to respond to, which is why one set of weights can caption and detect and segment. You're not loading a different model per job - you're changing which instruction you hand the same model.

That also explains the text_input box. For open-ended tasks like caption_to_phrase_grounding, referring_expression_segmentation or docvqa, whatever you type is what the model looks for or answers - "the dog", "the red car", "what is the total?". For the pure caption tasks (caption, detailed_caption, more_detailed_caption), leave it empty; the task token is the whole prompt.

The inputs and outputs that matter

You need three things wired in and one dropdown set:

• image - the picture to analyze.
• florence2_model (type FL2MODEL) - comes from a loader node (Florence2ModelLoader or the auto-downloading DownloadAndLoadFlorence2Model). Florence2Run can't load its own weights.
• task - the enum that decides everything. Start with more_detailed_caption for training captions, ocr for text, or caption_to_phrase_grounding to find a named thing.

text_input matters only for the grounding/segmentation/DocVQA tasks. The rest of the knobs are optional and fine at defaults: keep_model_loaded (turn it on if you're captioning a whole folder, so the model doesn't reload every run - costs VRAM but saves a lot of time), max_new_tokens (caption length cap, 1024 is generous), num_beams and do_sample (generation tuning - leave them unless output is repetitive), fill_mask, output_mask_select (pick a specific detected region when a task returns several), and seed.

Four outputs come back: caption (STRING - wire it into a CLIP Text Encode, a ShowText, or save it as the training .txt), image (the annotated preview with boxes/labels drawn on), mask (MASK, for the segmentation task → feeds inpainting or compositing), and data (raw JSON - the actual coordinates and labels, for downstream nodes that parse detections).

Common issues

The honest caveat, straight from users: it's great at captioning, less great at segmentation. When you ask referring_expression_segmentation for just a face or a hand, it frequently grabs the whole person instead. If you need tight, reliable masks, CLIPSeg or a SAM-based node is often the better call, and Florence-2 is best kept to the captioning and detection jobs it's genuinely good at.

Beyond that, most breakage is install-time (see the loader article): the pack needs transformers >= 4.38.0, and if the model dropdown upstream is empty you haven't downloaded weights yet. If captions come out empty or garbled, check the ComfyUI console - a version mismatch or a half-downloaded model shows up there first.