Joy_caption

This is the workhorse node of the whole pack: JoyCaption pre-alpha, the image-captioning VLM that became the community default for writing training captions. If you've seen people talking about "JoyCaption" for captioning Flux or SDXL datasets and then looked for it inside ComfyUI, this is what they mean. The README is Chinese, the author is StartHua, and the pack name (Comfyui_CXH_joy_caption) is a mouthful - but this one node is why anyone finds the pack at all.

The setup is two nodes, not one. Joy_caption_load builds the pipeline (SigLIP vision model + a 4-bit Llama 3.1 8B + the image adapter), and Joy_caption runs it on one image and hands you a string. It's a classic "load it, caption it" pair.

How it works

Under the hood this is the fancyfeast pre-alpha JoyCaption architecture, ported directly. Your image goes through Google's SigLIP vision encoder, which produces image features. Those features get pushed through a small MLP called the image adapter that maps them into Llama 3.1's embedding space. Then the image tokens are pasted in front of your text prompt, and the LLM generates a caption as if it were answering a normal chat.

That's the trick that made JoyCaption good: it's not a bespoke caption model, it's an 8B LLM with vision bolted on. Captions come out in natural language, and they're reliably better for Flux-style training than the old BLIP tags (BLIP is considered basically unusable by now - the community has moved on to JoyCaption and Florence 2).

The inputs that actually matter

You'll set prompt (default: "A descriptive caption for this image") and maybe max_new_tokens and temperature once. The one you really need to understand is cache:

• cache = False (default): after each caption, the node tears down the whole pipeline and frees VRAM. Your image-generation half of the workflow keeps its memory. Cost: the next caption reloads everything, so repeated single captions are slow.
• cache = True: keep the ~8–10 GB pipeline resident. Much faster for iterating on one image or captioning in a loop, but you'll fight for VRAM with whatever you're generating.

A common pattern: leave cache off when captioning one image in an otherwise-busy workflow, turn it on when you're churning through a batch.

Wiring it up

• Inputs: joy_pipeline (from Joy_caption_load), image (any IMAGE), prompt (multiline), max_new_tokens (10–4096, default 1024), temperature (0–1, default 0.7), cache.
• Output: one STRING - the caption. Wire it into a save-text node, a clip/text-encode, or feed it into a training pipeline. That's it; there's no image output.

Installation and the one file that trips everyone up

Install is the pack-standard two-step. Either use ComfyUI Manager (search "Comfyui_CXH_joy_caption") or:

cd ComfyUI/custom_nodes
git clone https://github.com/StartHua/Comfyui_CXH_joy_caption
# then, with ComfyUI's python:
cd ComfyUI/custom_nodes/Comfyui_CXH_joy_caption
python -m pip install -r requirements.txt

Restart ComfyUI. Note the README's warning: transformers cannot be too low - requirements pins transformers>=4.44.2.

Models auto-download to ComfyUI/models/ on first run: SigLIP to clip/, the 4-bit Llama to LLM/. But the image adapter must be downloaded by hand from the fancyfeast pre-alpha space (wpkklhc6) and dropped at models/Joy_caption/image_adapter.pt. Missing that file is the classic first-run error. The README also links a Baidu Pan mirror.

Troubleshooting

• OOM on a 8 GB card: you're not alone - the 4-bit Llama plus SigLIP is a lot. Use the unsloth/Meta-Llama-3.1-8B-bnb-4bit model option (the default) rather than full meta-llama, and keep cache off.
• The node never loads / error importing the pack: the code imports cv2 at module level but requirements.txt never installs OpenCV. If your ComfyUI environment is missing it, pip install opencv-python-headless fixes the import crash.
• Only runs on NVIDIA: several files hardcode "cuda". This pack won't run on AMD or Apple Silicon without editing source.