Extensions/ComfyUI-Gemma4-GGUF
ComfyUI Extension

ComfyUI-Gemma4-GGUF

ComfyUI custom node for image captioning using Gemma 4 GGUF models via llama-cpp-python. (Description by CC)

By promptlo·Created 5 months ago·Updated 5 months ago· 18
promptlo/ComfyUI-Gemma4-GGUF
Nodes—
On cloudLocal install
Stars18
Updated5 months ago
Readme

ComfyUI-Gemma4-GGUF

ComfyUI custom node for image captioning using Gemma 4 GGUF models via llama-cpp-python.

📺 Follow the author on Bilibili

中文文档

Features

  • Load GGUF-format Gemma 4 multimodal models (26B-A4B, 31B, E2B, E4B)
  • Uses Gemma4ChatHandler (official) or MTMDChatHandler for optimal image understanding
  • Configurable visual token budget (70/140/280/560/1120)
  • Auto VRAM-aware GPU layer calculation
  • 8 preset prompt styles (tags, simple, detailed, cinematic, OCR, etc.)
  • Advanced mode with full generation parameter control
  • Independent parameters node for reusable configs
  • Model unload support for memory management

Prerequisites

1. Install llama-cpp-python with Gemma 4 support

Gemma 4 requires the latest llama-cpp-python from source (PyPI releases do not include Gemma4ChatHandler).

NVIDIA GPU:

CMAKE_ARGS="-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=89" pip install --no-cache-dir git+https://github.com/JamePeng/llama-cpp-python.git

CUDA Architecture Reference: | GPU Series | Architecture | CMAKE_CUDA_ARCHITECTURES | |---|---|---| | RTX 50 series (Blackwell) | 12.0 | 120 | | RTX 40 series (Ada Lovelace) | 8.9 | 89 | | RTX 30 series (Ampere) | 8.6 | 86 | | RTX 20 series (Turing) | 7.5 | 75 | | GTX 10 series (Pascal) | 6.1 | 61 |

AMD GPU (ROCm):

CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install --no-cache-dir git+https://github.com/JamePeng/llama-cpp-python.git

macOS (Metal):

CMAKE_ARGS="-DGGML_METAL=on" pip install --no-cache-dir git+https://github.com/JamePeng/llama-cpp-python.git

CPU only:

pip install --no-cache-dir git+https://github.com/JamePeng/llama-cpp-python.git

Verify installation:

python -c "from llama_cpp.llama_chat_format import Gemma4ChatHandler; print('OK')"

2. Download GGUF Model Files

Download from unsloth/gemma-4-26B-A4B-it-GGUF and place in:

ComfyUI/models/LLM/Gemma4-GGUF/

Required files:

  • Main model: e.g., gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf (Dynamic 4-bit, recommended)
  • Multimodal projector: mmproj-BF16.gguf

Available quantizations: | Quantization | Size | Quality | |---|---|---| | UD-Q2_K_XL | ~8GB | Good | | UD-Q4_K_XL | ~14GB | Very Good (recommended) | | UD-Q5_K_M | ~16GB | Excellent | | UD-Q6_K | ~18GB | Near-lossless | | UD-Q8_K_XL | ~24GB | Best |

Installation

  1. Place this plugin in ComfyUI/custom_nodes/ComfyUI-Gemma4-GGUF/
  2. Install dependencies: 
    pip install -r ComfyUI/custom_nodes/ComfyUI-Gemma4-GGUF/requirements.txt
  3. Download GGUF model files to ComfyUI/models/LLM/Gemma4-GGUF/
  4. Restart ComfyUI

Nodes

Gemma4 GGUF Model Loader

Loads the GGUF model and multimodal projector.

Inputs:

  • model_file: Dropdown of available model GGUF files
  • mmproj_file: Dropdown of available mmproj GGUF files
  • vram_limit: VRAM limit in GB for auto GPU layer calculation (-1 = no limit)
  • n_gpu_layers: GPU layers to offload (-1 = all, overridden by vram_limit if > 0)
  • context_size: Context window size (default: 32768)
  • image_max_tokens: Visual token budget (70/140/280/560/1120)
  • image_min_tokens: Minimum visual tokens (0 = no minimum)
  • keep_model_loaded: Keep model in memory after inference

Outputs:

  • model: Loaded model bundle (GEMMA4_GGUF_MODEL type)

Gemma4 Image Caption

Generates a caption/description for an input image.

Inputs:

  • model: Loaded model from Model Loader
  • image: Input image tensor
  • system_prompt: System prompt to guide model behavior and language
  • preset_prompt: Choose from 8 preset prompt styles
  • user_prompt: Custom text prompt (overrides preset when filled)
  • max_tokens: Maximum output tokens
  • seed: Random seed

Outputs:

  • caption: Generated text description

Gemma4 Image Caption (Advanced)

Extended version with full generation parameter control.

Additional Inputs:

  • temperature: Sampling temperature (0.0-2.0, official default: 1.0)
  • top_p: Nucleus sampling (0.0-1.0, official default: 0.95)
  • top_k: Top-k sampling (1-200, official default: 64)
  • min_p: Minimum probability threshold (0.0-1.0, default: 0.05)
  • repeat_penalty: Repetition penalty (0.5-2.0, official default: 1.0)
  • present_penalty: Presence penalty (0.0-2.0, default: 0.0)
  • frequency_penalty: Frequency penalty (0.0-2.0, default: 0.0)
  • dry_multiplier: DRY repetition penalty (0.0-2.0, 0.0 = disabled, 0.8 = recommended for breaking loops)
  • dry_base: DRY exponential penalty base (1.0-3.0, default: 1.75)
  • dry_allowed_length: DRY max allowed repeat length (1-20, default: 2)

Gemma4 GGUF Parameters

Independent parameter node that outputs a GEMMA4_PARAMS object for reusable configs.

Gemma4 GGUF Unload Model

Manually unload the model from memory to free VRAM/RAM.

Visual Token Budget

Gemma 4 supports variable visual token budgets:

| Tokens | Use Case | |---|---| | 70 | Classification, fast captioning | | 140 | Basic captioning | | 280 | General multimodal (recommended) | | 560 | Charts, screenshots, UI reasoning | | 1120 | OCR, document parsing, handwriting |

Hardware Requirements

| Model | 4-bit | 8-bit | |---|---|---| | E2B | 4GB | 5-8GB | | E4B | 5.5-6GB | 9-12GB | | 26B-A4B | 16-18GB | 28-30GB | | 31B | 17-20GB | 34-38GB |

VRAM Planning for 16GB GPU (e.g., RTX 4060 Ti)

| Quantization | n_gpu_layers | n_ubatch | Notes | |---|---|---|---| | Q2_K_XL (~8GB) | -1 (all) | 2048 | Fits entirely in VRAM | | Q4_K_XL (~14GB) | 20-25 | 2048 | Partial GPU offload | | Q4_K_XL (~14GB) | -1 (all) | 2048 | Will OOM, do not use |

Example Workflow

  1. Add Gemma4 GGUF Model Loader node, select model and mmproj from dropdowns
  2. Set vram_limit to your GPU VRAM (e.g., 16) for auto layer calculation, or set n_gpu_layers manually
  3. Add Load Image node to load your image
  4. Add Gemma4 Image Caption node, connect model and image
  5. Select a preset prompt or write a custom one
  6. Run the workflow

Troubleshooting

"llama-cpp-python is not installed"

Install from source with the appropriate CMAKE_ARGS for your hardware (see Prerequisites).

"Failed to create a compatible chat handler"

Your llama-cpp-python version does not include Gemma4ChatHandler. Reinstall from source:

CMAKE_ARGS="-DGGML_CUDA=on" pip install --no-cache-dir git+https://github.com/JamePeng/llama-cpp-python.git

Out of memory

  • Use a lower quantization (Q2 instead of Q4)
  • Set vram_limit to a value below your total VRAM (e.g., 14 for a 16GB card)
  • Reduce context_size
  • Use the Unload node after inference

Model not found in dropdown

Place GGUF files in ComfyUI/models/LLM/Gemma4-GGUF/.

📺 Bilibili