hswq_Florence2Run

This is the node that does the work. Feed it an image and a loaded Florence-2 model, pick a task, and it comes back with text, an annotated image, a mask, and a JSON blob - whichever of those your task actually produces.

Florence-2 is a prompt-based seq2seq model: one set of weights, and the task is chosen by a token prepended to the input. That's why a single node covers captioning, detection, OCR, grounding and segmentation rather than one node per job. The task dropdown here is that token list.

The tasks, grouped by what they give you

Text out. caption, detailed_caption, more_detailed_caption - the three verbosity levels, and the ones you'll use 90% of the time. ocr reads text off the image. docvqa answers a question about a document. prompt_gen_tags, prompt_gen_mixed_caption, prompt_gen_analyze, prompt_gen_mixed_caption_plus are the PromptGen vocabulary - they only make sense with one of the MiaoshouAI PromptGen checkpoints loaded; on a plain Microsoft base they produce noise, not tags.

Boxes and an annotated image. region_caption, dense_region_caption, region_proposal, caption_to_phrase_grounding and ocr_with_region draw labelled boxes into the image output and fill white rectangles into the mask output.

Segmentation. referring_expression_segmentation returns polygon masks for the thing you describe - this is the task where the mask is a real shape rather than a rectangle.

Inputs and outputs

Required: image, florence2_model (FL2MODEL), text_input (multiline string), task, and fill_mask (default on - controls whether boxes/overlays are drawn).

Optional: keep_model_loaded (off by default - turn it on when you're captioning a folder and don't want the model reloading each run), max_new_tokens (1024), num_beams (1), do_sample (on), output_mask_select (empty), seed.

Outputs: image, mask, caption (STRING), data (JSON).

The field that will bite you

text_input is only legal for three tasks: referring_expression_segmentation, caption_to_phrase_grounding and docvqa. Put anything in it while running caption and the node raises outright - the check is explicit in the source, not a silent ignore. docvqa is the reverse: it requires text, and errors if the field is empty. So the rule is simple - leave it blank unless you're describing a region, grounding a phrase, or asking a document question.

Then read the outputs correctly:

• caption holds the text; for a single image it's a plain string, and for a batch it accumulates per-image results - test your downstream text node before trusting a batch.
• mask is filtered by output_mask_select, a comma-separated list of box indexes or labels (0,3, or a label name). Empty means all of them.
• data varies by task - bounding boxes for the region tasks, a label/box list for OCR-with-region.
• And the one everybody hits: for purely textual tasks (caption, ocr, docvqa, the prompt-gen family) there's no annotated image to produce, so image and mask come back as 64×64 black placeholders. Don't wire image into a Save Image node on a caption job; you'll save a tiny black square and wonder why.

On sampling: do_sample on with num_beams at 1 is the default and gives you variation per run (the seed is hashed into torch's RNG). Want deterministic output? Turn do_sample off and let beam search do the work.

Getting good captions out of it

The known limits are the ones every VLM has here. Multi-subject attribution goes wrong - with two people in frame, expect who-wears-what to get mixed - and a Florence caption describes a photo without being reliably reproducible from it. That's why the durable pattern is Florence for scene description plus a WD14 tagger for booru vocabulary, concatenated, and why hand-auditing a small training set still beats trusting the model.

One specific failure worth pre-empting: Florence-2 loves mentioning that an image is black and white or monochrome, and that token leaks straight into your positive prompt and drags the sampler toward grayscale. The same pack ships a Color Filter node for exactly that - Florence caption in, monochrome wording stripped, clean prompt out. If you're wiring a caption straight into a CLIP Text Encode, put something in between.

Installing it

Part of ussoewwin/ComfyUI-NunchakuFluxLoraStacker, whose Florence-2 nodes are a merged fork of kijai/ComfyUI-Florence2 - hence the hswq_ class prefix so both packs can coexist. Manager: search ComfyUI-NunchakuFluxLoraStacker. Manual:

cd ComfyUI/custom_nodes
git clone https://github.com/ussoewwin/ComfyUI-NunchakuFluxLoraStacker.git
pip install -r requirements.txt

Restart. You need transformers>=4.39.0,!=4.50.*, peft, accelerate, timm, matplotlib and pillow for this node (all in the requirements file), and a model in ComfyUI/models/LLM/ from one of the two loader nodes. Install the full requirements file: the pack imports its CCSR dependencies at load time, and a missing one removes every node in the pack, this one included.

If something's off

ValueError about text input. See above - blank it, or pick a task that takes text.

A 64×64 black image where you expected your picture. Expected on the text-only tasks.

Slow, or VRAM climbing across runs. keep_model_loaded is on. It's worth it for a folder of images, painful if you're also sampling at the same time.

Blank or garbage captions. Check the checkpoint/task pairing, especially prompt_gen_* tasks against a non-PromptGen model.