JoyCaption

If you're training a LoRA on an LLM-encoded base - Flux, Z-Image, Klein - your captions should read like natural language, not tag soup. The model the community actually reaches for there is JoyCaption, and this node is where you live while building that dataset: image in, caption out, repeat a thousand times.

JoyCaption is an 8B vision-language model (Llama 3.1 8B plus a SigLIP2 vision encoder in the LLaVA framework) that fpgaminer built specifically to write training captions, with equal SFW and NSFW coverage as the explicit design goal. The "uncensored" bit matters more than it sounds: the alternatives were an expensive, censored API or weaker open models that fell apart outside the SFW domain. This WO_JoyCaption node comes from the without-ordinary fork of fpgaminer's ComfyUI nodes, which is the version worth installing - it splits the model loader off so one loaded model can feed many caption runs, adds seed support, and lets you pick which device inference runs on (the original was hardcoded to "every GPU you have").

How it works

Feed it a loaded model (JOYCAPTIONMODEL from a JoyCaption Loader or Download And Load node) and an IMAGE. It assembles the actual query from your settings - the same prompt templates the model was trained against - runs the model, and hands you the caption.

Two details make this node better than the original. First, it returns both query and caption: the query is the literal text that went to the model, which is your debugging tool when a caption comes out wrong (all your extra options are baked in there, so you can see exactly what you asked). Second, it converts your ComfyUI image tensor back to a PIL image before inference, because JoyCaption was trained on Lanczos-resized images and the author doesn't trust a plain tensor resample to match.

The inputs that matter

The two dropdowns do most of the work:

• caption_type - twelve modes: Descriptive, Descriptive (Casual), Straightforward, Stable Diffusion Prompt, MidJourney, the booru tag lists (Danbooru, e621, Rule34, Booru-like), Art Critic, Product Listing, and Social Media Post. For training captions on a modern base, Descriptive or Straightforward is the sweet spot - the model's own docs call them the most useful. The tag modes exist, but for an Illustrious/Pony dataset a real WD14 tagger beats them.
• caption_length - "any", "very short" through "very long", or an exact word count from 20 to 260. Exact counts are handy when you want consistent caption lengths across a set.

Then the sampling knobs: max_new_tokens (512 default), temperature (0.6), top_p (0.9), top_k (0 = off), and seed. A temperature of 0 switches to greedy decoding. Two more that trip people up:

• keep_model_loaded - off by default, which means the model is offloaded and VRAM cleared after every run. Fine for one image. Miserable for a thousand.
• extra_options (optional) - plug in the output of the JoyCaption Extra Options node and its toggles get folded into the query.

Install

ComfyUI Manager → search wo_joycaption_comfyui (listed as "Without's JoyCaption ComfyUI Nodes") → install → restart. Or manually:

cd ComfyUI/custom_nodes
git clone https://github.com/without-ordinary/wo_joycaption_comfyui
# restart ComfyUI

The pack pulls in transformers, torch, torchvision, accelerate, bitsandbytes, pillow, and huggingface-hub. bitsandbytes only really matters for the 8-bit/4-bit memory modes. The model itself is a multi-gigabyte download on first run into models/LLavacheckpoints if you load it through the Download And Load node.

Common issues

The model's one honest weakness: like every VLM captioner in this space, JoyCaption mixes up who's doing what when two or more people are in frame. Auto-caption the bulk of your set, but audit multi-character images by hand - the LoRA training docs still call manual captioning the gold standard for small sets, and this is exactly why.

Quality and VRAM are the other classic trade-off. The 8-bit and 4-bit memory modes work, but the README is blunt that quality degrades. On a 12–24GB card, Default mode at fp16 is the sensible baseline; reach for quantization only when the diffusion model needs the VRAM too.