MinusZone - CLIPTextEncode(Florence-2)

CLIPTextEncode(Florence-2) trades the llama.cpp vision models for Microsoft's Florence-2, run through HuggingFace transformers. Feed it an image and it produces a natural-language caption describing the scene - then encodes it into conditioning if you wire in a clip. It's the in-graph flavor of the model the KB's lora-training guide singles out: Florence 2 is one of the community's go-to captioners for natural-language training captions, because it writes flowing sentences instead of the terse tag lists old BLIP-style models gave you.

There are two model choices, and they matter:

• Florence-2-large-ft (default) - the instruction-tuned "fine-tuned" variant. Better at following the captioning prompt, generally the pick for captioning.
• Florence-2-large - the base model, less instruction-friendly.

Both are loaded with AutoProcessor + AutoModelForCausalLM from transformers and run in fp16 on CUDA if available. First run downloads the model into ComfyUI/models/minus_zone_models - a multi-GB download, so budget for it.

The inputs that matter

• model_name - one of the two Florence-2 variants above.
• resolution - default 512. The image is resized before captioning. Florence-2 likes ~384-512 for captioning; bump toward 768 if fine details are getting missed, at the cost of speed.
• keep_device - False unloads the model after each run (frees VRAM for the diffusion model), True keeps it resident for fast repeat calls. If you're captioning in a loop, True is dramatically faster.

Optional: image (the IMAGE to caption) and clip (wired → you get CONDITIONING out).

Outputs: text (the caption) and conditioning (null if no clip).

How it works

The node loads Florence-2 via transformers, feeds it the resized image, and generates a caption with the model's default captioning prompt. The output is a natural-language sentence or two, not parenthesized keyword groups - so this is the "natural language caption for Flux-style training" path, not the A1111 tag path. When clip is wired, the caption is run through A1111-style weight-scaling encoding so you can use it directly as conditioning.

Installing

Part of ComfyUI-Prompt-MZ - ComfyUI Manager, search "ComfyUI-Prompt-MZ", or:

cd ComfyUI/custom_nodes
git clone https://github.com/MinusZoneAI/ComfyUI-Prompt-MZ

The extra dependency is transformers (plus torch, which you already have). ComfyUI ships with a transformers-capable environment on most installs; if the node errors on import, pip install transformers into the ComfyUI venv.

Troubleshooting

Two predictable rough spots. First, the model download is big and goes to minus_zone_models - a slow or interrupted first run looks like a hang; watch the console for progress, and a retry resumes rather than restarting if the pack's diskcache is intact. Second, VRAM: Florence-2-large is ~1.5B params and fp16 pushes a few GB of usage. On an 8GB card with keep_device off it's fine, but keep keep_device off so it unloads before the sampler needs its own VRAM - otherwise you get OOM right at the sampler step. And a fair warning: the default caption prompt produces descriptive sentences, which is what you want for Flux-style datasets and useless for tag-based (Danbooru/Illustrious) training - for that, pair the pack with a WD14 tagger instead, or use customize_instruct-style rewriting elsewhere.