JoyCaption GGUF (Advanced)

If you're bulk-captioning a folder of images for a LoRA and the plain JoyCaption GGUF node keeps giving you captions that are too flowery, too short, or just not quite what you asked for, this is the node that lets you actually argue with it. Same model, same quantized GGUF weights, but every generation parameter is exposed: temperature, top-p, top-k, token budget, and a raw custom prompt field. It's the difference between "caption this image" and "caption this image, my way."

JoyCaption itself was built by fpgaminer (fancyfeast on Hugging Face) as part of the bigASP SDXL finetune project, specifically because the existing options for training captions were bad: ChatGPT is expensive and censored, CogVLM is weak outside SFW content, and BLIP's formulaic captions have been called "pure trash" by the community for years. Free, open weights, equal coverage of SFW and NSFW - that's why it's become the community favorite for captioning datasets that feed LLM-encoder models like Flux, Z-Image, Qwen-Image, and Krea 2. This GGUF build is 1038lab's packaging of it for ComfyUI, quantized via llama.cpp so it doesn't eat your whole VRAM budget just to write a sentence.

How it works

Under the hood this runs through llama-cpp-python rather than the raw HuggingFace transformers stack the non-GGUF JoyCaption nodes use - the model is quantized into one of a dozen-plus precision tiers (Q2_K up through F16), and llama.cpp's engine loads whichever tier you picked, feeds it your image plus a prompt built from prompt_style and caption_length (or your own custom_prompt), and generates text token by token. That last part is what temperature, top-p, and top-k actually control.

The inputs and outputs that matter

• image - what you're captioning, obviously.
• model - which GGUF quant to load (IQ4_XS is the default and the README's recommended sweet spot; Q2_K if you're VRAM-starved, Q8_0/F16 if you have room and want the highest fidelity).
• prompt_style - don't sleep on this one. Beyond Descriptive/Straightforward, it covers Danbooru tag list, e621 tag list, Rule34 tag list, Stable Diffusion Prompt, MidJourney, Art Critic, and Product Listing. Pick the one that matches what you're actually training toward.
• temperature, top_p, top_k - the sampling trio. 1038lab's own guidance: keep temperature around 0.6-0.7 for stable, repeatable captions, and only push toward 0.8-1.0 for more varied phrasing. top_k at 0 disables it; 40-50 tightens the model onto more predictable word choices.
• custom_prompt - write your own instruction instead of a preset, if none of the twelve styles quite fit.
• memory_management - Global Cache (24GB+ VRAM) keeps the model resident for fast batch runs, Keep in Memory (12GB+) is the balanced default, Clear After Run (8GB+) frees VRAM after every image at the cost of reload time.

There's also an optional extra_options socket (type JOYCAPTION_EXTRA_OPTIONS) for a companion options node if you have one; otherwise leave it disconnected.

Two string outputs: STRING is the generated caption, the one you wire into a text writer or a CLIP text encode. PROMPT is the assembled instruction actually sent to the model - useful for sanity-checking what prompt_style + custom_prompt produced before trusting it to a few thousand images.

How to install it

Search "ComfyUI-JoyCaption" in ComfyUI Manager, or manually:

cd ComfyUI/custom_nodes
git clone https://github.com/1038lab/ComfyUI-JoyCaption.git
cd ComfyUI-JoyCaption
pip install -r requirements.txt

Then, because this is the GGUF node specifically, you need llama-cpp-python with CUDA support - the step people skip and then wonder why nothing loads. The pack ships an automated installer (python llama_cpp_install/llama_cpp_install.py) that handles CUDA detection for you, or you can pip install -r requirements_gguf.txt manually. Download a GGUF model plus its vision-projection (mmproj) file from the README's table and drop them in ComfyUI/models/LLM/GGUF - or just run the node and let it auto-download and rename the default model on first use.

Common issues & troubleshooting

ModuleNotFoundError: No module named 'llama_cpp'. The single most common failure across every ComfyUI node that wraps llama-cpp-python, not just this one - it means the package didn't install into your actual ComfyUI Python environment, or installed without CUDA wheels. Run the automated installer script rather than a bare pip install llama-cpp-python; it exists specifically to dodge this.

It loads but runs on CPU and is painfully slow. Check processing_mode actually landed on GPU (Auto should pick it if available) and confirm your llama-cpp-python build was compiled with CUDA - the manual requirements_gguf.txt path can silently hand you a CPU-only wheel.

Captions are repetitive or generic across a whole folder. That's temperature and top_k doing exactly what they're set to do. Nudge temperature up slightly for more variety, or write a more specific custom_prompt instead of relying on a generic preset.

Out of memory partway through a batch. Drop to a smaller quant (Q4_K_M or lower) before you drop memory_management to Clear After Run - quant size affects every run, memory mode only affects idle VRAM between them.