JoyCaption Run

JoyCaption Run is the local-captioning half of this pack: it runs the community's default natural-language captioner - fpgaminer's JoyCaption - on your own GPU, no API, no key, no uploads. If you're building a LoRA training set for a Flux/Z-Image-style base, this is the node you actually want from the whole pack, because it's the one that does the heavy, private, offline job the API nodes can't.

JoyCaption is the 8B vision-language model that became the training-caption standard (joycaption.json has the full story): built from scratch because the alternatives were an expensive censored API or a weak open model, with equal SFW and NSFW coverage as the founding design goal. This node wraps it for ComfyUI with the practical extras - batch-caption a whole folder of images, save each caption next to its image, and pick an NF4-quantized build that fits on modest VRAM. For a large dataset it's the difference between hand-writing hundreds of captions and walking away while the card churns.

How it works

The node loads a JoyCaption model from your ComfyUI/models/LLM/ folder into memory (a cached singleton, so repeated runs reuse it), runs inference over either a single image tensor or every image in an images_dir folder, and writes/returns captions. The "nf4" model variants load 4-bit via bitsandbytes - the README calls it "a blessing for low VRAM users," and it is. unload_model (default on) frees the VRAM after each run so the model doesn't squat on your card between jobs. The heavy dependencies - transformers, bitsandbytes, accelerate, optimum - are all in the pack's requirements.txt.

The inputs that matter

• image (IMAGE, optional) - a single image from the canvas.
• images_dir (STRING, optional) - a folder path; the node finds every image in it and captions them in a batch, saving each caption to a .txt next to its image (or into save_img_prompt_to_folder if you set that). This is the node's superpower.
• preset_prompt - ten one-shot caption styles (formal, casual, stable diffusion prompt, MidJourney, booru tags, art critic, product listing, social post...). Leave it on the default for descriptive captions, or set it to None - important, see below.
• merge_prompt (BOOLEAN) - looks like it appends your custom prompt to the preset. It doesn't, in practice.
• model - four choices: alpha-two / beta-one, each in full or NF4. llama-joycaption-beta-one-hf-llava-nf4 (default) is the current best on consumer VRAM.
• max_new_tokens / temperature / top_k / top_p / do_sample / use_cache / seed - generation controls; defaults are sensible.
• unload_model - leave on unless you're chaining many caption runs back to back.
• tagger (optional) - an extra tag string prepended to the prompt for the booru-style modes.

Output: text (STRING) - the caption for a single image, or a confirmation path for folder batches.

Installing it

Part of ComfyUI_Prompt-All-In-One (billwuhao). ComfyUI Manager → search Prompt-All-In-One, or clone + pip install -r requirements.txt. The models do not auto-download - grab the one you want from the README's HuggingFace links (fancyfeast for the full builds, John6666 for the NF4 quants) and place the folder under ComfyUI/models/LLM/. Restart, and it's under 🎤MW/MW-Prompt-All-In-One.

Where people get burned

Two gotchas from reading the source. First, preset_prompt overrides your prompt whenever it isn't None, and the merge_prompt toggle doesn't actually merge - the merged string gets overwritten by the preset on the next line of code. So if your custom prompt isn't taking effect, set preset_prompt to None. Second, NF4 needs a working bitsandbytes install - on Windows that means the right prebuilt wheel for your torch/CUDA version; if the model fails to load, that's the first place to look. Also remember JoyCaption's known weakness (it's in llm-in-comfyui.md): multi-subject attribution is shaky, so audit anything important by hand. But for the bulk job - hundreds of images, natural-language captions, offline and uncensored - this node is the real deal.