ComfyUI-JoyCaption

Joy Caption is a ComfyUI custom node powered by the LLaVA model for efficient, stylized image captioning. Caption Tools nodes handle batch image processing and automatic separation of caption text.

News & Updates

• 2025/06/07: Update ComfyUI-JoyCaption to v1.1.1 ( update.md )

• 2025/06/05: ComfyUI-JoyCaption to v1.1.0 ( update.md )

• Add Caption tools

Features

• Simple and user-friendly interface
• Multiple caption types support
• Flexible length control
• Memory optimization options
• Automatic model download - the model will be downloaded automatically and properly renamed on first use
• Support for multiple caption types
• Support for advanced customization options
• Multiple precision options (fp32, bf16, fp16)

Installation

1. Clone this repository to your ComfyUI/custom_nodes directory:

cd ComfyUI/custom_nodes
git clone https://github.com/1038lab/ComfyUI-JoyCaption.git

2. Install the required dependencies:

cd ComfyUI/custom_nodes/ComfyUI-JoyCaption
pip install -r requirements.txt

Download Models

The models will be automatically downloaded and renamed on first use, or you can manually download them: | Model | Link | | ----- | ---- | | JoyCaption Beta One | Download | | JoyCaption Alpha Two | Download |

After downloading, place the model files in your ComfyUI/models/LLM/ directory.

Basic Usage

Basic Node

1. Add the "JoyCaption" node from the 🧪AILab/📝JoyCaption category
2. Connect an image source to the node
3. Select the model file (defaults to llama-joycaption-beta-one-hf-llava)
4. Adjust the parameters as needed
5. Run the workflow

Advanced Node

1. Add the "JoyCaption (Advanced)" node from the 🧪AILab/📝JoyCaption category
2. Connect an image source to the node
3. Select the caption type
4. Adjust the parameters as needed
5. Run the workflow

Parameters

Basic Node

| Parameter | Description | Default | Range | | --------- | ----------- | ------- | ----- | | Model | The JoyCaption model to use | llama-joycaption-beta-one-hf-llava | - | | Memory Control | Memory optimization settings | Default | Default (fp32), Balanced (8-bit), Maximum Savings (4-bit) | | Caption Type | Caption style selection | Descriptive | Descriptive, Descriptive (Casual), Straightforward, Tags, Technical, Artistic | | Caption Length | Output length control | medium | any, very short, short, medium, long, very long |

Quantization Options

| Mode | Precision | Memory Usage | Speed | Quality | Recommended GPU | |------|-----------|--------------|-------|---------|----------------| | Default | fp32 | ~16GB | 1x | Best | 24GB+ | | Default | bf16 | ~8GB | 1.5x | Excellent | 16GB+ | | Default | fp16 | ~8GB | 2x | Very Good | 16GB+ | | Balanced | 8-bit | ~4GB | 2.5x | Good | 12GB+ | | Maximum Savings | 4-bit | ~2GB | 3x | Acceptable | 8GB+ |

Advanced Node

| Parameter | Description | Default | Range | | --------- | ----------- | ------- | ----- | | Extra Options | Additional feature options | [] | Multiple options | | Person Name | Name for person descriptions | "" | Any text | | Max New Tokens | Maximum tokens to generate | 512 | 1-2048 | | Temperature | Generation temperature | 0.6 | 0.0-2.0 | | Top-p | Sampling parameter | 0.9 | 0.0-1.0 | | Top-k | Top-k sampling | 0 | 0-100 | | Precision | Use fp32 for best quality, bf16 for balanced performance, fp16 for maximum speed. 8-bit and 4-bit quantization provide significant memory savings with minimal quality impact |

Setting Tips

| Setting | Recommendation | | ------- | -------------- | | Memory Mode | Based on GPU VRAM: 24GB+ use Default, 12GB+ use Balanced, 8GB+ use Maximum Savings | | Input Resolution | The model works best with images of 512x512 or higher resolution | | Memory Usage | If you encounter memory issues, try using 4-bit mode or processing images at a lower resolution | | Performance | For batch processing, consider reducing max_new_tokens for faster throughput | | Temperature | Lower values (0.6-0.7) for more stable results, higher values (0.8-1.0) for more diverse results |

About Model

This implementation uses LLaVA-based image captioning models, optimized to provide fast and accurate image descriptions.

Model features:

• Free and open weights
• Uncensored content coverage
• Broad style diversity (digital art, photoreal, anime, etc.)
• Fast processing with bfloat16 precision
• High-quality caption generation
• Memory-efficient operation
• Consistent results across various image types
• Support for multiple caption styles
• Optimized for training diffusion models

The models are trained on diverse datasets, ensuring:

• Balanced representation across different image types
• High accuracy in various scenarios
• Robust performance with complex scenes
• Support for both SFW and NSFW content
• Equal coverage of different styles and concepts

Roadmap

Future plans include:

• Support for more caption types
• Additional optimization options
• Improved memory management
• Batch processing support

Credits

• JoyCaption Model: fancyfeast
• Created by: 1038lab

License

This repository's code is released under the GPL-3.0 License.