ialhabbal VLLM GGUF Advanced

ialhabbal_VLLM_GGUF_Advanced is the same local-GGUF Qwen/Gemma node as its plain sibling, with every llama.cpp runtime knob dragged onto the canvas. The plain GGUF node is "download two files, describe the image." This one is for when the default context window is too small, the model won't fit on the card, or generation is crawling and you want to know which knob is to blame.

Everything about the model side is identical: the same nine-file dropdown (Gemma and Qwen, heavily abliterated), the same local-only files that you must place under ComfyUI/models/llm/GGUF/, the same mmproj vision projector requirement, the same optional image/video inputs and single RESPONSE string output. What's new is control.

The llama.cpp knobs, decoded

• gpu_layers (default -1) - how many layers to offload to the GPU. -1 means all of them, which is what you want if the model fits. Set it to 0 for pure CPU. If you're splitting VRAM with a diffusion model, try something like 20–30 and let the rest ride on CPU. This is the single most important knob for VRAM budgeting.
• ctx (default 8192, up to 262144) - the context window. Bigger context = more room for long image sequences and long answers, but the KV cache eats memory and grows with it. 8192 is plenty for captioning a single image; raise it for chatty multi-image workflows and expect the RAM/VRAM bill.
• n_batch (default 512) - tokens processed per batch during prompt ingestion. Higher = faster prompt processing, up to the point where your hardware saturates.
• image_max_tokens (default 4096, up to ~1M) - the token budget for image embeddings. More tokens per image = finer visual detail understood, but each one costs time and memory. 4096 is the sane default for a single photo.
• pool_size (default 4194304 bytes ≈ 4 MB) - llama.cpp's scratch-pool for the vision pipeline. Leave it alone unless you're seeing vision-related allocation errors; the default exists because it works.
• top_k (default 0) - top-k sampling. 0 means disabled (llama.cpp treats it as "no cap"). If you enable it, 40–100 is the usual range; combined with top_p it tightens up the output.
• temperature, top_p, repetition_penalty, frame_count, device - the same sampling and video controls as the HF Advanced node, with a wider temperature range (up to 2.0) since llama.cpp will take it.

device behaves like the HF node: auto, cpu, mps, or a specific cuda:N on multi-GPU boxes.

When you'd use this instead of the plain GGUF node

Almost always, honestly - the defaults mirror the plain node's behavior exactly, so you lose nothing by having the knobs visible. The one case for the simple node is if you never want to look at a pool_size field in your life and trust the defaults. For everyone else, this is the version to keep in your workflow: same setup, no extra cost, and when generation misbehaves you can actually do something about it.

Setup and gotchas

Same install as the whole pack (ComfyUI Manager → ComfyUI-ialhabbal, or git clone https://github.com/ialhabbal/ComfyUI-ialhabbal.git into custom_nodes, then restart). The llama-cpp-python requirement from the README:

pip install --upgrade pip setuptools wheel
pip install "llama-cpp-python[server]"

The recurring failure modes carry straight over from the plain GGUF node: files missing from models/llm/GGUF/, the mmproj half-forgotten, and mismatched model/projector pairs. New ones to watch here:

• OOM after raising ctx - the KV cache isn't free. Drop ctx back or lower gpu_layers.
• Vision errors around pool_size - bump it in ~1 MB steps; some Gemma models want more scratch than the default.
• Slow even with gpu_layers -1 - Q8_K_P is a big quant. The Q5_K_M or Q4_K_M entries are the practical speed/quality tradeoffs, exactly as the GGUF ladder in general says: Q8 is basically fp16, and Q4_K_M is where 12 GB cards live.