🐰 Florence2Run /v (from Kijai)

Florence2Run is the workhorse half of the Florence-2 pair that MaraScott vendored from Kijai's ComfyUI-Florence2. If you've ever wanted a single node that can caption an image, read the text in it, draw boxes around objects, or turn a region into a mask - without touching a ControlNet or a separate captioner - this is it. It's the node behind a lot of "auto-caption for training" and "describe what's in this image" workflows, and it beats the old CLIP-captioner days by a mile.

You feed it an image and a loaded Florence-2 model, pick a task, and it returns whatever that task produces: a caption string, a mask, an annotated image, or structured JSON.

How it works

Florence-2 is a Microsoft vision-language model - a sequence-to-sequence transformer that takes an image plus a task prompt and generates a text answer, which can then be converted back into images and masks. This node is Kijai's implementation, bundled in verbatim, so the mechanism is exactly what the ComfyUI-Florence2 pack does: the model's processor encodes your image, the model generates tokens (max_new_tokens, num_beams, do_sample, seed all behave like any HF generation settings), and the result is post-processed into the four outputs depending on the task.

It needs a Florence-2 model loaded first - either DownloadAndLoadFlorence2Model (also in this pack, under MaraScott/vendor) or the original from Kijai's own pack. The model itself is a FL2MODEL type, and it downloads from Hugging Face on first use into ComfyUI/models/LLM.

The inputs and outputs that matter

• image - the image to analyze.
• florence2_model - the loaded model from the loader node.
• text_input - free text used by prompt-dependent tasks like docvqa or referring_expression_segmentation ("the red car").
• task - the big dropdown, and the thing you'll actually change. 15 choices including caption, detailed_caption, more_detailed_caption, ocr, region_proposal, dense_region_caption, referring_expression_segmentation, and the PromptGen family. caption and detailed_caption are where most people start.
• fill_mask - when on, regions from grounding-style tasks come back as a filled MASK.
• Optional: keep_model_loaded, max_new_tokens (default 1024), num_beams (3), do_sample (on), output_mask_select, seed.

Outputs: image (the annotated image with boxes/labels drawn), mask (segmentation mask when the task produces one), caption (the generated text), and data (JSON with the raw results - boxes, polygons, OCR text).

Installing it

Part of the MaraScott pack: ComfyUI Manager → search "MaraScott" → install → restart. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/MaraScott/ComfyUI_MaraScott_Nodes

Then restart ComfyUI. The model is the real install: pick microsoft/Florence-2-base (small) or -large (better, slower, fatter), and it downloads automatically. Give it a few GB of disk. The pack pins transformers==4.37.2 in its requirements - fine for Florence-2, but it can collide with other packs that need a newer transformers, so install this pack and any other LLM pack in the same session and be ready to reconcile versions.

Common issues

The model will unload between runs unless keep_model_loaded is on - leave it on if you're chaining several tasks or doing a big batch. flash_attention_2 requires extra deps and a compatible GPU; if the loader errors, switch attention to sdpa on the loader, which is the safe default anyway. And do_sample with a low seed makes captions a little nondeterministic - for stable outputs in a pipeline, turn sampling off or fix the seed.