hswq_DownloadAndLoadFlorence2Model

Florence-2 is the captioner you reach for when JoyCaption is more than you need: Microsoft's prompt-based vision model, 0.23B parameters in the base size and 0.77B in large, MIT-licensed, and small enough to run on a T4. It reached ComfyUI through Kijai's node within days of the June 2024 weights, and it stuck because one tiny model does captioning, detailed captioning, detection, grounding, OCR and segmentation - you switch behaviour with a task token, not a different checkpoint.

This node is the "get me the weights and load them" half of that in this pack. Pick a repo from a dropdown, queue something, and it downloads into ComfyUI/models/LLM/<repo name>/. That's the whole pitch: no manual Hugging Face visit, no git lfs in a terminal.

The preset list, briefly

The dropdown covers the useful families - microsoft/Florence-2-base and -base-ft (the default, fastest), Florence-2-large / -large-ft (0.77B, better captions, still small), HuggingFaceM4/Florence-2-DocVQA for documents, the thwri/CogFlorence-2.1/2.2-Large community fine-tunes, gokaygokay/Florence-2-SD3-Captioner and Florence-2-Flux-Large for caption styles aimed at those bases, the four MiaoshouAI/Florence-2-*-PromptGen-v1.5/v2.0 builds, and PJMixers-Images/Florence-2-base-Castollux-v0.5. It's a curated list on purpose: anything off it raises a value error rather than guessing at a config.

If you want a checkpoint that isn't here - your own fine-tune, a random HF repo - that's what the sibling hswq_Florence2ModelLoader node is for, which loads directories already sitting in models/LLM.

Inputs and outputs

• model - the repo dropdown, default microsoft/Florence-2-base.
• precision - fp16 (default), bf16, fp32. fp16 is the right default; bf16 is a good idea on 30-series and newer if fp16 gives you oddities; fp32 doubles the download footprint in RAM for no captioning benefit.
• attention - flash_attention_2, sdpa (default), eager, sage_attention_2, sage_attention_3. Leave it on sdpa unless you've installed what the others need: Flash Attention 2 needs the flash-attn package, SageAttention 2 wants the sageattention package and a 3090/4090-or-newer card, and SageAttention 3 needs Blackwell (5090) plus the sageattn3 wheel.
• optional lora - a PEFTLORA input, which is where the pack's hswq_DownloadAndLoadFlorence2Lora plugs in.
• optional convert_to_safetensors - the author's tooltip explains it: older weights aren't saved as .safetensors and load more slowly, so this converts .bin → .safetensors. Be aware it then deletes the original .bin on success - fine, but it does mutate your model folder.
• output florence2_model (FL2MODEL) → straight into hswq_Florence2Run.

What happens when you hit queue

The first run is when it downloads, so it looks frozen: gigabytes going into ComfyUI/models/LLM/ with console lines and no progress bar worth watching. After that the node picks its loader path by your transformers version. On transformers 5.x it uses the pack's own loader - building the model under init_empty_weights() and streaming tensors in from model.safetensors (or pytorch_model.bin) - which is the path maintained for the newer dtype/PreTrainedModel APIs. On older transformers it falls back to a classic from_pretrained. Sage attention modes are set on the config as SDPA and swapped in at the module level, and they only engage on transformers ≥ 5.0; if you're below that you'll see a printed warning and silently get SDPA instead.

The model lands on ComfyUI's offload device and is moved to the GPU per run, with hswq_Florence2Run pushing it back afterwards unless you enable its keep_model_loaded option. That's why a captioning pass doesn't permanently eat your VRAM.

Installing it

Part of ussoewwin/ComfyUI-NunchakuFluxLoraStacker, which merged a fork of kijai/ComfyUI-Florence2 into itself under nodes/florence2/ - the hswq_ class prefix exists so these nodes can coexist with Kijai's original pack without class-type collisions. Manager: search ComfyUI-NunchakuFluxLoraStacker. Manual:

cd ComfyUI/custom_nodes
git clone https://github.com/ussoewwin/ComfyUI-NunchakuFluxLoraStacker.git
pip install -r requirements.txt

Restart. The Florence-2 half needs transformers>=4.39.0,!=4.50.*, peft, accelerate, timm, matplotlib and pillow - all in the pack's requirements, along with the CCSR upscaler's dependencies, which are imported at load time and will take the whole pack down if they're missing. You also need network access on the first run so huggingface_hub can fetch the repo.

If something's off

It says the download finished but the run errors with "No model weights found". You got a partial folder - interrupted download, no disk space, or an offline queue. Delete ComfyUI/models/LLM/<model-name> and queue again.

ValueError: Model X is not in the supported model list. You're asking this node for a repo that isn't one of the fourteen presets. Use the local-directory loader instead.

An error about flash attention or sage attention imports. The attention backend you chose needs a package you don't have. Set attention back to sdpa and everything works.

It's slow for its size. Florence-2 is the fast option here - if you're comparing against JoyCaption, remember that's an 8B model. If it feels slow, check you're not on large when base would do, and that you're not double-loading the model by wiring keep_model_loaded on and leaving another Florence node in the graph.