# UPDATE: Tencent has released official Comfy ELLA nodes that should now be used instead of this: https://github.com/TencentQQGYLab/ComfyUI-ELLA # ComfyUI wrapper nodes to use the Diffusers implementation of ELLA ![image](https://github.com/kijai/ComfyUI-ELLA-wrapper/assets/40791699/d255385f-dd10-4eec-ada7-b7774080c6a7) ![image](https://github.com/kijai/ComfyUI-ELLA-wrapper/assets/40791699/2d97a745-accc-4ff1-a0c4-c37481ce732f) # Installing Either use the Manager and it's install from git -feature, or clone this repo to custom_nodes and run: `pip install -r requirements.txt` or if you use portable (run this in ComfyUI_windows_portable -folder): `python_embeded\python.exe -m pip install -r ComfyUI\custom_nodes\ComfyUI-ELLA-wrapper\requirements.txt` You can use any 1.5 model, rest is **auto downloaded**, sources: ELLA model (132MB): https://huggingface.co/QQGYLab/ELLA/blob/main/ella-sd1.5-tsc-t5xl.safetensors Also requires google-flan-t5-xl, for which I have opted to autodownload this repo, 6GB): https://huggingface.co/ybelkada/flan-t5-xl-sharded-bf16/tree/main # Original repo: # ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment

Xiwei Hu*, Rui Wang*, Yixiao Fang*, Bin Fu*, Pei Cheng, Gang Yu✦

* Equal contributions, ✦ Corresponding Author

Official code of "ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment".

## 🌟 Changelog - **[2024.4.9]** 🔥🔥🔥 Release [ELLA-SD1.5](https://huggingface.co/QQGYLab/ELLA/blob/main/ella-sd1.5-tsc-t5xl.safetensors) Checkpoint! Welcome to try! - **[2024.3.11]** 🔥 Release DPG-Bench! Welcome to try! - **[2024.3.7]** Initial update ## Inference ### ELLA-SD1.5 ```bash # get ELLA-SD1.5 at https://huggingface.co/QQGYLab/ELLA/blob/main/ella-sd1.5-tsc-t5xl.safetensors # comparing ella-sd1.5 and sd1.5 # will generate images at `./assets/ella-inference-examples` python3 inference.py test --save_folder ./assets/ella-inference-examples --ella_path /path/to/ella-sd1.5-tsc-t5xl.safetensors # build a demo for ella-sd1.5 GRADIO_SERVER_NAME=0.0.0.0 GRADIO_SERVER_PORT=8082 python3 ./inference.py demo /path/to/ella-sd1.5-tsc-t5xl.safetensors ``` ## 📊 DPG-Bench The guideline of DPG-Bench: 1. Generate your images according to our [prompts](./dpg_bench/prompts/). It is recommended to generate 4 images per prompt and grid them to 2x2 format. **Please Make sure your generated image's filename is the same with the prompt's filename.** 2. Run the following command to conduct evaluation. ```bash bash dpg_bench/dist_eval.sh $YOUR_IMAGE_PATH $RESOLUTION ``` Thanks to the excellent work of [DSG](https://github.com/j-min/DSG) sincerely, we follow their instructions to generate questions and answers of DPG-Bench. ## 📝 TODO - [ ] add huggingface demo link - [x] release checkpoint - [x] release inference code - [x] release DPG-Bench ## 💡 Others We have also found [LaVi-Bridge](https://arxiv.org/abs/2403.07860), another independent but similar work completed almost concurrently, which offers additional insights not covered by ELLA. The difference between ELLA and LaVi-Bridge can be found in [issue 13](https://github.com/ELLA-Diffusion/ELLA/issues/13). We are delighted to welcome other researchers and community users to promote the development of this field. ## 😉 Citation If you find **ELLA** useful for your research and applications, please cite us using this BibTeX: ``` @misc{hu2024ella, title={ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment}, author={Xiwei Hu and Rui Wang and Yixiao Fang and Bin Fu and Pei Cheng and Gang Yu}, year={2024}, eprint={2403.05135}, archivePrefix={arXiv}, primaryClass={cs.CV} } ```

Name	Type	Description
model	MODEL	—
clip	CLIP	—
vae	VAE	—

Diffusers Model Loader

Inputs (3)

Outputs (1)