Nodes/ComfyUI_Simple_Qwen3-VL-gguf/🌐 LLM Config (Advanced)
ComfyUI Node

🌐 LLM Config (Advanced)

Every knob on the local Qwen3-VL loader, finally behind one node

By KLL535Β·Created 9 months agoΒ·Updated about 13 hours agoΒ· 85
🌐 LLM Config (Advanced)
    • config
    • diff_config
    β—„model_presetNoneβ–Ί
    β—„πŸ“ Model & Pathstrueβ–Ί
    β—„model_pathβ–Ί
    β—„mmproj_pathβ–Ί
    β—„πŸ—„οΈ Memory & Contexttrueβ–Ί
    β—„n_ctx8192β–Ί
    β—„n_batch2048β–Ί
    β—„n_ubatch512β–Ί
    β—„n_keep256β–Ί
    β—„offload_kqvtrueβ–Ί
    β—„type_k1=F16β–Ί
    β—„type_v1=F16β–Ί
    β—„use_mmapfalseβ–Ί
    β—„use_mlockfalseβ–Ί
    β—„pool_size4194304β–Ί
    β—„logits_allfalseβ–Ί
    β—„ctx_checkpoints0β–Ί
    β—„swa_fullfalseβ–Ί
    β—„πŸŽ² Sampling & Generationtrueβ–Ί
    β—„max_tokens2048β–Ί
    β—„temperature0.70β–Ί
    β—„top_p0.92β–Ί
    β—„min_p0.05β–Ί
    β—„top_k0β–Ί
    β—„repeat_penalty1.10β–Ί
    β—„presence_penalty0.00β–Ί
    β—„frequency_penalty0.00β–Ί
    β—„enable_thinkingfalseβ–Ί
    β—„force_reasoningfalseβ–Ί
    β—„words_to_banβ–Ί
    β—„βš™οΈ Hardware & Accelerationtrueβ–Ί
    β—„n_gpu_layers-1β–Ί
    β—„n_cpu_moe0β–Ί
    β—„cpu_moefalseβ–Ί
    β—„n_threads8β–Ί
    β—„flash_attn_type-1=AUTOβ–Ί
    β—„split_mode0-NONEβ–Ί
    β—„main_gpu0β–Ί
    β—„cuda_deviceβ–Ί
    β—„tensor_splitβ–Ί
    β—„πŸ’¬ Chat, Prompts & Variablesfalseβ–Ί
    β—„chat_handlernoneβ–Ί
    β—„chat_formatnoneβ–Ί
    β—„chat_format_from_gguffalseβ–Ί
    β—„system_prompt_defaultβ–Ί
    β—„system_preset_to_user_promptfalseβ–Ί
    β—„user_prompt_after_contenttrueβ–Ί
    β—„enable_variablesfalseβ–Ί
    β—„add_vision_idautoβ–Ί
    β—„add_image_idβ–Ί
    β—„add_frame_idβ–Ί
    β—„add_audio_idβ–Ί
    β—„πŸ“ Prompt Templatefalseβ–Ί
    β—„raw_modefalseβ–Ί
    β—„prompt_templateβ–Ί
    β—„stopβ–Ί
    β—„πŸ–ΌοΈ Multimodal & Mediafalseβ–Ί
    β—„force_mmprojtrueβ–Ί
    β—„image_min_tokens0β–Ί
    β—„image_max_tokens0β–Ί
    β—„max_images10β–Ί
    β—„max_frames24β–Ί
    β—„max_audios3β–Ί
    β—„audio_sample_rate0β–Ί
    β—„image_quality95β–Ί
    β—„frame_quality75β–Ί
    β—„πŸ”’ Embeddingsfalseβ–Ί
    β—„extract_embeddingfalseβ–Ί
    β—„pooling_type0-NONEβ–Ί
    β—„tokenizer_pathβ–Ί
    β—„embedding_scale1.0β–Ί
    β—„convert_emb_to_condfalseβ–Ί
    β—„πŸ› οΈ Debug, System & Advancedfalseβ–Ί
    β—„verbosefalseβ–Ί
    β—„debugtrueβ–Ί
    β—„debug_outputfalseβ–Ί
    β—„raw_outputfalseβ–Ί
    β—„clearing_cachetrueβ–Ί
    β—„force_gc_startfalseβ–Ί
    β—„force_gc_unloadfalseβ–Ί
    β—„scriptqwen3vl_run.pyβ–Ί
    β—„extraβ–Ί
    β—„config_overrideβ€”β–Ί

    If you've ever hand-edited a JSON config to get an LLM node to work, this is the node that retires that hobby. 🌐 LLM Config (Advanced) is the configurator for KLL535's SimpleQwenVL pack - the one that loads GGUF vision-language models (Qwen3-VL, Qwen3.5/3.6, Gemma4, MiniCPM…) locally through llama.cpp, no API and no key. The pack's core node takes its entire setup as a JSON string; this node is a friendly GUI for writing that string, with all 73+ llama.cpp parameters laid out instead of buried in a config file.

    How it works

    The node doesn't run the model. It builds the config and hands it to the main Simple Qwen-VL Vision Language Model node as a single config STRING output. You wire config into the core node's config_override input and the loader takes it from there.

    Parameters are grouped into nine collapsible sections - Model & Paths, Memory & Context, Sampling & Generation, Hardware & Acceleration, Chat & Prompts, Prompt Template, Multimodal & Media, Embeddings, and Debug - so by default you only see the sensible defaults and expand what you actually need. When the node runs, it merges in a strict priority order: base config β†’ whatever your widget values say β†’ your config_override text input applied last. That override is the sneaky good part: it's stackable and can be plain-ish text, because json_repair will fix missing commas and quotes for you.

    There's also built-in preset management. Save, rename, and delete presets right from the node - they land in ComfyUI/user/SimpleQwenVL_configs/system_prompts_user.json, and whatever you save appears in the model_preset dropdown on both this node and the core node. Set up a model once, then pick it from a list forever.

    The inputs that actually matter

    The full list is huge, but a beginner sets a handful:

    • model_path and mmproj_path - the GGUF weights and the multimodal projector. mmproj_path is required for vision models; on Windows the Browse buttons find them for you (the file dialogs are Windows-only via ctypes, so on Linux/macOS you type paths).
    • chat_handler - picks the chat template (qwen3, qwen35, gemma4, llava15…). Required for vision models; get this wrong and the model talks gibberish.
    • n_ctx - context size. The tooltip gives you the rule: image_tokens + input_tokens + max_tokens ≀ n_ctx. Bigger context eats more VRAM.
    • n_gpu_layers - -1 puts everything on GPU, 0 is CPU-only. If you OOM, step it down.
    • n_cpu_moe - for MoE models, keeps that many experts on CPU instead of letting the GPU spill into shared memory. The pack author's whole "make a 35B fit in 16GB" trick.
    • max_tokens + enable_thinking - thinking models (Qwen3, Gemma4) burn through output tokens and emit <think>…</think> blocks you'll want stripped (the pack's Simple Remove Think node does that).

    Installing it

    Install the pack via ComfyUI Manager (search "ComfyUI_Simple_Qwen3-VL-gguf") or clone it:

    cd ComfyUI/custom_nodes
git clone https://github.com/KLL535/ComfyUI_Simple_Qwen3-VL-gguf

    Restart ComfyUI and refresh the frontend. Then the real work: the stock pip install llama-cpp-python is stale and doesn't support Qwen3 at all. You need JamePeng's fork - 0.3.17+ for qwen3-VL, 0.3.30+ for qwen3.5, 0.3.35+ for gemma4. Either grab a prebuilt WHL from his releases or, as the README recommends, build from source:

    set CMAKE_ARGS=-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=89 -DCMAKE_BUILD_TYPE=Release
python -m pip install .

    (89 is a 40-series card - see the README table for yours.) Prebuilt WHLs may lack AVX/VMM acceleration, which matters for n_cpu_moe. The pack itself also pulls json_repair, colorama, pillow, opencv-python, and av.

    Troubleshooting

    • ggml_new_object: not enough space - bump pool_size (the tooltip suggests 8MB/16MB) or lower n_ctx.
    • Failed to load shared library ... ggml.dll - missing CUDA runtime. ComfyUI's PyTorch usually provides it; otherwise install the CUDA Toolkit and match driver versions. The README's issue #15 covers this in detail.
    • Sudden 5-7x slowdown - your VRAM overflowed into shared memory. Watch the CUDA graph in Task Manager; fix with n_cpu_moe, KV-cache quantization (type_k/type_v), or a stronger quant.

    Note the author tests on Windows (RTX 5080/2060, Python 3.13). This node is brand new and only surfaced in the pack's nightly updates, so if something's janky, that's probably why - turn on debug in the Debug group and run in subprocess mode to get clean logs.

    Category🌐 SimpleQwenVL

    Inputs (83)

    NameTypeDefaultDescription
    model_presetCOMBONoneSelect model preset
    πŸ“ Model & PathsBOOLEANtrueShow/hide group: model and projector paths.
    model_pathSTRINGPath to GGUF model file. Relative paths are supported (relative to custom_nodes dir).
    mmproj_pathSTRINGPath to multimodal projector file. Required for vision models.
    πŸ—„οΈ Memory & ContextBOOLEANtrueShow/hide group: context, batches, memory pool, KV cache.
    n_ctxINT8192512–1048576Context size. Rule: image_tokens + input_tokens + max_tokens <= n_ctx.
    n_batchINT204832–65536Batch size for prompt processing. Lower = less VRAM, higher = faster.
    n_ubatchINT51232–65536Micro-batch size for advanced memory management.
    n_keepINT2560–131072Number of tokens to keep in KV-cache from the initial prompt. Useful for few-shot / long-context scenarios.
    offload_kqvBOOLEANtrueOffload KV Cache to GPU. Turn OFF to save VRAM (will be slower).
    type_kCOMBO1=F16KV-cache quantization type for K. Some variants may not work.
    type_vCOMBO1=F16KV-cache quantization type for V. Some variants may not work.
    use_mmapBOOLEANfalseEnable mmap. On Windows it is often better to turn it off.
    use_mlockBOOLEANfalseEnable mlock. Lock model in RAM to prevent OS swapping.
    pool_sizeINT41943040–104857600Memory pool size for llama.cpp. Increase if you get 'ggml_new_object: not enough space'. 0 = default.
    logits_allBOOLEANfalseIf True, llama.cpp evaluates logits for ALL tokens (not only the last one). Required for perplexity evaluation and some scoring tasks, but significantly increases VRAM and time.
    ctx_checkpointsINT00–4096Max number of context checkpoints to create per slot. 0 = disabled.
    swa_fullBOOLEANfalseEnable full Sliding Window Attention context. Required for some models to prevent truncation.
    🎲 Sampling & GenerationBOOLEANtrueShow/hide group: generation limits and sampling parameters.
    max_tokensINT204816–131072Maximum number of tokens to generate. Thinking models usually need more.
    temperatureFLOAT0.700–2Sampling temperature. Lower = deterministic, higher = creative.
    top_pFLOAT0.920–1Nucleus sampling cumulative probability cutoff.
    min_pFLOAT0.050–1Minimum probability for a token to be considered (filters out unlikely tokens).
    top_kINT00–500Limit to top-K most likely tokens. 0 disables top-k filtering.
    repeat_penaltyFLOAT1.100–3Penalty for repeating tokens. Values >1 discourage repetition.
    presence_penaltyFLOAT0.00-2–2Penalty based on token presence. Positive values encourage new topics.
    frequency_penaltyFLOAT0.00-2–2Penalty based on token frequency. Positive values reduce repetition.
    enable_thinkingBOOLEANfalseEnable thinking/reasoning process (for Gemma, Qwen, MiniCPM, GLM, etc.).
    force_reasoningBOOLEANfalseFor Qwen3: force reasoning mode even on simple queries.
    words_to_banSTRINGComma-separated list of banned words. Applies logit_bias of -100 to their tokens.
    βš™οΈ Hardware & AccelerationBOOLEANtrueShow/hide group: GPU layers, MoE offload, threads, multi-GPU.
    n_gpu_layersINT-1-1–999Number of layers to offload to GPU. -1 = all, 0 = CPU only.
    n_cpu_moeINT00–128For MoE models: number of experts to keep on CPU. Saves VRAM. Slower than full GPU offload, but faster and more stable than letting the OS swap when VRAM is overcommitted.
    cpu_moeBOOLEANfalseFor MoE models: unload ALL experts into RAM. Minimal VRAM usage.
    n_threadsINT81–256Number of CPU threads to use for inference (and prompt processing).
    flash_attn_typeCOMBO-1=AUTOFlash Attention backend for llama.cpp. Requires a compatible build.
    split_modeCOMBO0-NONEGPU splitting mode: 0=NONE, 1=LAYER, 2=ROW.
    main_gpuINT00–16Index of the primary GPU when split_mode=NONE.
    cuda_deviceSTRINGSets CUDA_VISIBLE_DEVICES before init. Single index or comma-separated list.
    tensor_splitSTRINGFractions of the model to offload to each GPU (split_mode=LAYER).
    πŸ’¬ Chat, Prompts & VariablesBOOLEANfalseShow/hide group: chat handlers, formats, system prompts, and variables.
    chat_handlerCOMBOnoneChat handler for multimodal models.
    chat_formatCOMBOnoneChat format for text-only models.
    chat_format_from_ggufBOOLEANfalseForce loading chat template from the GGUF metadata. Note: Does not work with images/audio/video.
    system_prompt_defaultSTRINGDefault system prompt for the model.
    system_preset_to_user_promptBOOLEANfalseMove the system preset from the system prompt role to the user prompt role.
    user_prompt_after_contentBOOLEANtrueInsert user_prompt after the image/audio/video content. False = before.
    enable_variablesBOOLEANfalseEnable substitution of {placeholders} in system and user prompts.
    add_vision_idCOMBOautoAdd vision ID token. 'auto' = script decides (True if images != 1 or video > 0).
    add_image_idSTRINGTemplate to label images before insertion. {num} = image index.
    add_frame_idSTRINGTemplate to label video frames before insertion. {num} = frame index.
    add_audio_idSTRINGTemplate to label audio clips before insertion. {num} = audio index.
    πŸ“ Prompt TemplateBOOLEANfalseShow/hide group: custom raw prompt templates and stop sequences.
    raw_modeBOOLEANfalseEnable custom raw prompt template mode (bypasses chat handlers).
    prompt_templateSTRINGCustom prompt template. Must include {system}, {images}, {user}.
    stopSTRINGStop sequences. JSON list: ["</s>", "[INST]"] or comma-separated. Empty = use handler defaults.
    πŸ–ΌοΈ Multimodal & MediaBOOLEANfalseShow/hide group: image/audio/video limits and quality.
    force_mmprojBOOLEANtrueLoad mmproj even without media inputs (preserves template for enable_thinking).
    image_min_tokensINT00–65536Minimum tokens for image embeddings. 0 = not set.
    image_max_tokensINT00–65536Maximum tokens for image embeddings. 0 = not set.
    max_imagesINT101–100Limit on the total number of incoming images.
    max_framesINT241–512Limit on video frames. More frames require larger context.
    max_audiosINT31–100Limit on the number of incoming audio clips.
    audio_sample_rateINT00–192000Target sampling frequency for audio resampling. 0 = not set.
    image_qualityINT951–100JPEG quality (1-100) when encoding images to data URIs.
    frame_qualityINT751–100JPEG quality (1-100) when encoding video frames to data URIs.
    πŸ”’ EmbeddingsBOOLEANfalseShow/hide group: embedding extraction mode (replaces text generation).
    extract_embeddingBOOLEANfalseSwitch node to embedding mode. Uses LlamaEmbedding. Text output is replaced by a CONDITIONING tensor.
    pooling_typeCOMBO0-NONEPooling strategy for LlamaEmbedding. NONE = no pooling (per-token embeddings), MEAN = average pool, CLS = use [CLS] token, LAST = use last token.
    tokenizer_pathSTRINGPath to external HuggingFace tokenizer. Overrides built-in llama.cpp tokenizer.
    embedding_scaleFLOAT1.00.1–1000Scalar multiplier applied to the output embedding vector. 1.0 = no scaling. Useful to match the magnitude expected by downstream models.
    convert_emb_to_condBOOLEANfalseWrap the raw embedding into a ComfyUI CONDITIONING structure (hidden_states + attention_mask). Required for passing embeddings into SD/Flux conditioning slots.
    πŸ› οΈ Debug, System & AdvancedBOOLEANfalseShow/hide group: logging, garbage collection, cache, and advanced settings.
    verboseBOOLEANfalseEnables verbose logging from llama.cpp.
    debugBOOLEANtrueEnables timing output for each stage to the console.
    debug_outputBOOLEANfalsePrint the final LLM text output to console.
    raw_outputBOOLEANfalseIf True, disables output.strip() (keeps leading/trailing whitespaces).
    clearing_cacheBOOLEANtrueClear cache to prevent execution freezing during heavy memory activity.
    force_gc_startBOOLEANfalseForce garbage collection after memory clearing when unload_all_models is active.
    force_gc_unloadBOOLEANfalseForce garbage collection after deleting the LLM model.
    scriptSTRINGqwen3vl_run.pyName of the Python script to execute.
    extraSTRINGJSON dict of extra keys passed to the backend script.
    config_overrideoptSTRINGStackable config override (JSON or plain text). Applied last, highest priority.

    Outputs (2)

    NameTypeDescription
    configSTRINGβ€”
    diff_configSTRINGβ€”