ComfyUI Extension: ComfyUI LLM SDXL Adapter
A comprehensive set of ComfyUI nodes for using Large Language Models (LLM) as text encoders for SDXL image generation through a trained adapter.
Custom Nodes (0)
README
ComfyUI LLM SDXL Adapter
A comprehensive set of ComfyUI nodes for using Large Language Models (LLM) as text encoders for SDXL image generation through a trained adapter.
<img width="1643" alt="image" src="https://github.com/user-attachments/assets/f1517b7a-2497-4eea-9034-802b3469f172" />š Key Features
- Multi-LLM Support: Gemma and other compatible models (extensible architecture)
- Flexible Architecture: Modular nodes for customizable workflows
- Transformer Adapter: Convert LLM embeddings to SDXL format
- Chat Templates: Support for system prompts and conversation formatting
- Memory Optimization: Model loading management and caching
- Configurable Parameters: Flexible adapter architecture configuration
š¦ Installation
Requirements
- Python 3.8+
- ComfyUI
- CUDA (recommended)
- Latest transformers library (tested on 4.53.1)
Install Dependencies
pip install transformers>=4.53.1 safetensors einops torch
Install Nodes
- Clone the repository to
ComfyUI/custom_nodes/
:
cd ComfyUI/custom_nodes/
git clone https://github.com/NeuroSenko/ComfyUI_LLM_SDXL_Adapter.git
- Restart ComfyUI
šļø Project Structure
comfyui_llm_sdxl_adapter/
āāā __init__.py # Entry point and node registration
āāā llm_model_loader.py # LLM model loader
āāā llm_text_encoder.py # LLM text encoder
āāā lll_adapter_loader.py # Adapter loader
āāā llm_to_sdxl_adapter.py # Adapter architecture
āāā apply_llm_to_sdxl_adapter.py # Adapter application
āāā utils.py # Utility functions
āāā README.md # Documentation
š§ ComfyUI Nodes
LLM Model Loader
Loads language model and tokenizer.
Inputs:
model_path
: Path to LLM modeldevice
: Device (auto/cuda:0/cuda:1/cpu)force_reload
: Force reload model
Outputs:
model
: LLM model instancetokenizer
: Tokenizerinfo
: Model information
LLM Text Encoder
Encodes text using loaded LLM.
Inputs:
model
: LLM modeltokenizer
: Tokenizertext
: Text to encodesystem_prompt
: System prompt (optional)skip_first
: Number of tokens to skip
Outputs:
hidden_states
: LLM hidden statesinfo
: Encoding information
LLM Adapter Loader
Loads trained LLM -> SDXL adapter.
Inputs:
adapter_path
: Path to adapter file (.safetensors)llm_dim
: LLM dimension (default 1152)sdxl_seq_dim
: SDXL sequence dimension (2048)sdxl_pooled_dim
: SDXL pooled output dimension (1280)target_seq_len
: Target sequence length (308)n_wide_blocks
: Number of wide blocks (2)n_narrow_blocks
: Number of narrow blocks (3)
Outputs:
adapter
: Adapter instanceinfo
: Adapter information
Apply LLM To SDXL Adapter
Applies adapter to LLM hidden states.
Inputs:
llm_hidden_states
: LLM hidden statesadapter
: Loaded adapter
Outputs:
conditioning
: SDXL conditioninginfo
: Application information
šÆ Basic Workflow
LLM Model Loader ā LLM Text Encoder ā LLM Adapter Loader ā Apply LLM To SDXL Adapter ā KSampler
Usage Example:
- Load Model: Use
LLM Model Loader
to load an LLM (e.g., Gemma-3-1b-it) - Encode Text: Connect model to
LLM Text Encoder
and input your prompt - Load Adapter: Use
LLM Adapter Loader
to load trained adapter - Apply Adapter: Connect hidden states and adapter through
Apply LLM To SDXL Adapter
- Generate: Connect the resulting conditioning to standard SDXL pipeline
š File Structure
LLM Models
Place LLM models in the folder:
ComfyUI/models/LLM/
āāā gemma-3-1b-it/
Adapters
Place trained adapters in the folder:
ComfyUI/models/llm_adapters/
āāā gemma_to_sdxl_adapter.safetensors
āļø Adapter Architecture
The LLMToSDXLAdapter
uses a transformer architecture with two stages:
- Wide Processing: Process full sequence (512 tokens)
- Compression: Compress to target length (308 tokens) via cross-attention
- Narrow Processing: Final processing of compressed sequence
- Pooling: Create pooled output for SDXL
Key Features:
- Positional embeddings for input and output sequences
- Learnable compression queries for controlled compression
- Attention masking for proper padding handling
- Dropout and LayerNorm for training stability
š Debugging
To enable detailed logging:
import logging
logging.basicConfig(level=logging.INFO)
Information about loaded nodes is displayed during ComfyUI initialization.