ymc_node_joy

🤖 comfyui custom nodes to caption image with joy

## Why - use joy to caption image for aigc. - use joy to caption image files in diretory to do sth. (eg. lora training) ## Features ## Nodes - nodes show in console:

- nodes show in right mouse menu:

## Install ```bash # cd to comfyui/custom_nodes git clone https://github.com/ymc-github/ymc_node_joy ``` - **deps will be installed automatically** if deps in requirements.txt were not installed when comfyui up ## Required Models Essential components: - `google/siglip-so400m-patch14-384` (Vision model) - `unsloth/Meta-Llama-3.1-8B-bnb-4bit` or `meta-llama/Meta-Llama-3.1-8B` (LLM) - `Joy_caption/image_adapter.pt` (Custom adapter) ```plaintext / ├── models/ │ ├── clip/ # SigLIP Vision Model │ │ └── siglip-so400m-patch14-384/ │ ├── llm/ # Llama Language Model │ │ ├── Meta-Llama-3.1-8B-bnb-4bit/ │ │ └── Meta-Llama-3.1-8B/ │ └── Joy_caption/ # Custom Components │ └── image_adapter.pt # Dimension Adapter ``` ### 1. SigLIP Vision Model (google/siglip-so400m-patch14-384) **International**: https://huggingface.co/google/siglip-so400m-patch14-384 **China Mirror**: https://hf-mirror.com/google/siglip-so400m-patch14-384 ### 2. Llama Language Models #### 4bit Quantized (unsloth/Meta-Llama-3.1-8B-bnb-4bit) **International**: https://huggingface.co/unsloth/Meta-Llama-3.1-8B-bnb-4bit **China Mirror**: https://hf-mirror.com/unsloth/Meta-Llama-3.1-8B-bnb-4bit #### Original (meta-llama/Meta-Llama-3.1-8B) **International**: https://huggingface.co/meta-llama/Meta-Llama-3.1-8B (Access approval required) **China Mirror**: https://hf-mirror.com/meta-llama/Meta-Llama-3.1-8B ### 3. Image Adapter (Joy_caption/image_adapter.pt) **International**: https://huggingface.co/spaces/fancyfeast/joy-caption-pre-alpha/tree/main/wpkklhc6 **China Mirror**: https://www.modelscope.cn/models/fireicewolf/joy-caption-pre-alpha/files ## Usage - you can find it in search box : double click + typing keywords (eg: `joy`, `caption`) - you can find it in right mouse menu : `ymc/caption` - ~~you can find it in right mouse menu : `utils/ymc/caption` (as alias)~~ ## Demo - workflow demo:

## Based-on - pypi package [yors_comfyui_node_setup](https://pypi.org/project/yors_comfyui_node_setup/) - setup comfyui custom nodes easily - pypi package [yors_pano_ansi_color](https://pypi.org/project/yors_pano_ansi_color/) - info msg in console with color in your comfyui custom nodes easily - ~~pypi package [yors_pano_zero_field](https://pypi.org/project/yors_pano_zero_field/) - set nodes input field to be HQ in your comfyui custom nodes easily~~ ## Published to Comfy registry - get more details in [publish_to_comfy.yml](.github/workflows/publish_to_comfy.yml) - [docs for publishing to comfy registey](https://docs.comfy.org/registry/overview) - installed with comfy-cli ? `comfy node registry-install ymc_node_joy` ## Author name|email|desciption :--|:--|:-- yemiancheng||Main developer and code maintainer| chenxinghua|<455758525@qq.com>|Code reference from [StartHua/Comfyui_CXH_joy_caption](https://github.com/StartHua/Comfyui_CXH_joy_caption)| ## License MIT

joy model loader

Inputs (1)

Outputs (1)