Nodes/ComfyUI_Simple_Qwen3-VL-gguf/🌐 LLM Config
ComfyUI Node

🌐 LLM Config

Every knob on the local Qwen3-VL loader, finally behind one node

By KLL535Β·Created 11 months agoΒ·Updated 8 days agoΒ· 91
🌐 LLM Config
    • config
    • diff_config
    β—„model_presetNoneβ–Ί
    β—„πŸ“ Model & Pathstrueβ–Ί
    β—„model_pathβ–Ί
    β—„mmproj_pathβ–Ί
    β—„πŸ—„οΈ Memory & Contexttrueβ–Ί
    β—„n_ctx8192β–Ί
    β—„n_batch2048β–Ί
    β—„n_ubatch512β–Ί
    β—„n_keep256β–Ί
    β—„offload_kqvtrueβ–Ί
    β—„type_k1=F16β–Ί
    β—„type_v1=F16β–Ί
    β—„use_mmapfalseβ–Ί
    β—„use_mlockfalseβ–Ί
    β—„pool_size4194304β–Ί
    β—„logits_allfalseβ–Ί
    β—„swa_fullfalseβ–Ί
    β—„πŸŽ² Sampling & Generationtrueβ–Ί
    β—„max_tokens2048β–Ί
    β—„temperature0.70β–Ί
    β—„top_p0.92β–Ί
    β—„min_p0.05β–Ί
    β—„top_k0β–Ί
    β—„repeat_penalty1.10β–Ί
    β—„presence_penalty0.00β–Ί
    β—„frequency_penalty0.00β–Ί
    β—„enable_thinkingfalseβ–Ί
    β—„remove_thinkingfalseβ–Ί
    β—„answer_delimiterβ–Ί
    β—„force_reasoningfalseβ–Ί
    β—„words_to_banβ–Ί
    β—„βš™οΈ Hardware & Accelerationtrueβ–Ί
    β—„n_gpu_layers-1β–Ί
    β—„n_cpu_moe0β–Ί
    β—„cpu_moefalseβ–Ί
    β—„n_threads8β–Ί
    β—„flash_attn_type-1=AUTOβ–Ί
    β—„split_mode0=NONEβ–Ί
    β—„main_gpu0β–Ί
    β—„cuda_deviceβ–Ί
    β—„tensor_splitβ–Ί
    β—„πŸ’¬ Chat, Prompts & Variablesfalseβ–Ί
    β—„chat_handlernoneβ–Ί
    β—„chat_formatnoneβ–Ί
    β—„chat_format_from_gguffalseβ–Ί
    β—„system_prompt_defaultβ–Ί
    β—„system_preset_to_user_promptfalseβ–Ί
    β—„user_prompt_after_contenttrueβ–Ί
    β—„enable_variablesfalseβ–Ί
    β—„add_vision_idautoβ–Ί
    β—„add_image_idβ–Ί
    β—„add_frame_idβ–Ί
    β—„add_audio_idβ–Ί
    β—„πŸ“ Prompt Templatefalseβ–Ί
    β—„raw_modefalseβ–Ί
    β—„prompt_templateβ–Ί
    β—„stopβ–Ί
    β—„πŸ–ΌοΈ Multimodal & Mediafalseβ–Ί
    β—„force_mmprojtrueβ–Ί
    β—„image_min_tokens0β–Ί
    β—„image_max_tokens0β–Ί
    β—„max_images10β–Ί
    β—„max_frames24β–Ί
    β—„max_audios3β–Ί
    β—„audio_sample_rate0β–Ί
    β—„image_quality95β–Ί
    β—„frame_quality75β–Ί
    β—„video_modeimagesβ–Ί
    β—„video_fps_target1.0β–Ί
    β—„video_timestamp_interval_ms5000β–Ί
    β—„mmproj_batch_max_tokens1024β–Ί
    β—„βš‘ Speculative Decodingfalseβ–Ί
    β—„speculative_enabledfalseβ–Ί
    β—„speculative_type3=MTP (Multi-token Prediction)β–Ί
    β—„draft_n_max2β–Ί
    β—„draft_p_min0.00β–Ί
    β—„draft_model_pathβ–Ί
    β—„draft_n_gpu_layers-1β–Ί
    β—„draft_backend_samplingtrueβ–Ί
    β—„ngram_size_n8β–Ί
    β—„ngram_size_m16β–Ί
    β—„ngram_min_hits1β–Ί
    β—„ngram_max_entries_per_key4β–Ί
    β—„ctx_checkpoints0β–Ί
    β—„checkpoint_on_devicefalseβ–Ί
    β—„πŸ”’ Embeddings & TTSfalseβ–Ί
    β—„extract_embeddingfalseβ–Ί
    β—„pooling_type0=NONEβ–Ί
    β—„tokenizer_pathβ–Ί
    β—„embedding_scale1.0β–Ί
    β—„convert_emb_to_condfalseβ–Ί
    β—„extract_ttsfalseβ–Ί
    β—„mmproj_use_gputrueβ–Ί
    β—„mmproj_flash_attntrueβ–Ί
    β—„languageβ–Ί
    β—„πŸ› οΈ Debug, System & Advancedfalseβ–Ί
    β—„verbosefalseβ–Ί
    β—„debugtrueβ–Ί
    β—„debug_outputfalseβ–Ί
    β—„raw_outputfalseβ–Ί
    β—„streaming_modefalseβ–Ί
    β—„clearing_cachetrueβ–Ί
    β—„force_gc_startfalseβ–Ί
    β—„force_gc_unloadfalseβ–Ί
    β—„scriptqwen3vl_run.pyβ–Ί
    β—„extraβ–Ί
    β—„config_overrideβ€”β–Ί

    If you've ever hand-edited a JSON config to get an LLM node to work, this is the node that retires that hobby. 🌐 LLM Config (Advanced) is the configurator for KLL535's SimpleQwenVL pack - the one that loads GGUF vision-language models (Qwen3-VL, Qwen3.5/3.6, Gemma4, MiniCPM…) locally through llama.cpp, no API and no key. The pack's core node takes its entire setup as a JSON string; this node is a friendly GUI for writing that string, with all 73+ llama.cpp parameters laid out instead of buried in a config file.

    How it works

    The node doesn't run the model. It builds the config and hands it to the main Simple Qwen-VL Vision Language Model node as a single config STRING output. You wire config into the core node's config_override input and the loader takes it from there.

    Parameters are grouped into nine collapsible sections - Model & Paths, Memory & Context, Sampling & Generation, Hardware & Acceleration, Chat & Prompts, Prompt Template, Multimodal & Media, Embeddings, and Debug - so by default you only see the sensible defaults and expand what you actually need. When the node runs, it merges in a strict priority order: base config β†’ whatever your widget values say β†’ your config_override text input applied last. That override is the sneaky good part: it's stackable and can be plain-ish text, because json_repair will fix missing commas and quotes for you.

    There's also built-in preset management. Save, rename, and delete presets right from the node - they land in ComfyUI/user/SimpleQwenVL_configs/system_prompts_user.json, and whatever you save appears in the model_preset dropdown on both this node and the core node. Set up a model once, then pick it from a list forever.

    The inputs that actually matter

    The full list is huge, but a beginner sets a handful:

    • model_path and mmproj_path - the GGUF weights and the multimodal projector. mmproj_path is required for vision models; on Windows the Browse buttons find them for you (the file dialogs are Windows-only via ctypes, so on Linux/macOS you type paths).
    • chat_handler - picks the chat template (qwen3, qwen35, gemma4, llava15…). Required for vision models; get this wrong and the model talks gibberish.
    • n_ctx - context size. The tooltip gives you the rule: image_tokens + input_tokens + max_tokens ≀ n_ctx. Bigger context eats more VRAM.
    • n_gpu_layers - -1 puts everything on GPU, 0 is CPU-only. If you OOM, step it down.
    • n_cpu_moe - for MoE models, keeps that many experts on CPU instead of letting the GPU spill into shared memory. The pack author's whole "make a 35B fit in 16GB" trick.
    • max_tokens + enable_thinking - thinking models (Qwen3, Gemma4) burn through output tokens and emit <think>…</think> blocks you'll want stripped (the pack's Simple Remove Think node does that).

    Installing it

    Install the pack via ComfyUI Manager (search "ComfyUI_Simple_Qwen3-VL-gguf") or clone it:

    cd ComfyUI/custom_nodes
git clone https://github.com/KLL535/ComfyUI_Simple_Qwen3-VL-gguf

    Restart ComfyUI and refresh the frontend. Then the real work: the stock pip install llama-cpp-python is stale and doesn't support Qwen3 at all. You need JamePeng's fork - 0.3.17+ for qwen3-VL, 0.3.30+ for qwen3.5, 0.3.35+ for gemma4. Either grab a prebuilt WHL from his releases or, as the README recommends, build from source:

    set CMAKE_ARGS=-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=89 -DCMAKE_BUILD_TYPE=Release
python -m pip install .

    (89 is a 40-series card - see the README table for yours.) Prebuilt WHLs may lack AVX/VMM acceleration, which matters for n_cpu_moe. The pack itself also pulls json_repair, colorama, pillow, opencv-python, and av.

    Troubleshooting

    • ggml_new_object: not enough space - bump pool_size (the tooltip suggests 8MB/16MB) or lower n_ctx.
    • Failed to load shared library ... ggml.dll - missing CUDA runtime. ComfyUI's PyTorch usually provides it; otherwise install the CUDA Toolkit and match driver versions. The README's issue #15 covers this in detail.
    • Sudden 5-7x slowdown - your VRAM overflowed into shared memory. Watch the CUDA graph in Task Manager; fix with n_cpu_moe, KV-cache quantization (type_k/type_v), or a stronger quant.

    Note the author tests on Windows (RTX 5080/2060, Python 3.13). This node is brand new and only surfaced in the pack's nightly updates, so if something's janky, that's probably why - turn on debug in the Debug group and run in subprocess mode to get clean logs.

    Category🌐 SimpleQwenVL

    Inputs (107)

    NameTypeDefaultDescription
    model_presetCOMBONoneSelect model preset
    πŸ“ Model & PathsBOOLEANtrueShow/hide group: model and projector paths.
    model_pathSTRINGPath to GGUF model file. Relative paths are supported (relative to custom_nodes dir).
    mmproj_pathSTRINGPath to multimodal projector file. Required for vision models.
    πŸ—„οΈ Memory & ContextBOOLEANtrueShow/hide group: context, batches, memory pool, KV cache.
    n_ctxINT8192512–1048576Context size. Rule: image_tokens + input_tokens + max_tokens <= n_ctx.
    n_batchINT204832–65536Batch size for prompt processing. Lower = less VRAM, higher = faster.
    n_ubatchINT51232–65536Micro-batch size for advanced memory management.
    n_keepINT2560–131072Number of tokens to keep in KV-cache from the initial prompt. Useful for few-shot / long-context scenarios.
    offload_kqvBOOLEANtrueOffload KV Cache to GPU. Turn OFF to save VRAM (will be slower).
    type_kCOMBO1=F16KV-cache quantization type for K. Some variants may not work.
    type_vCOMBO1=F16KV-cache quantization type for V. Some variants may not work.
    use_mmapBOOLEANfalseEnable mmap. On Windows it is often better to turn it off.
    use_mlockBOOLEANfalseEnable mlock. Lock model in RAM to prevent OS swapping.
    pool_sizeINT41943040–104857600Memory pool size for llama.cpp. Increase if you get 'ggml_new_object: not enough space'. 0 = default.
    logits_allBOOLEANfalseIf True, llama.cpp evaluates logits for ALL tokens (not only the last one). Required for perplexity evaluation and some scoring tasks, but significantly increases VRAM and time.
    swa_fullBOOLEANfalseEnable full Sliding Window Attention context. Required for some models to prevent truncation.
    🎲 Sampling & GenerationBOOLEANtrueShow/hide group: generation limits and sampling parameters.
    max_tokensINT204816–131072Maximum number of tokens to generate. Thinking models usually need more.
    temperatureFLOAT0.700–2Sampling temperature. Lower = deterministic, higher = creative.
    top_pFLOAT0.920–1Nucleus sampling cumulative probability cutoff.
    min_pFLOAT0.050–1Minimum probability for a token to be considered (filters out unlikely tokens).
    top_kINT00–500Limit to top-K most likely tokens. 0 disables top-k filtering.
    repeat_penaltyFLOAT1.100–3Penalty for repeating tokens. Values >1 discourage repetition.
    presence_penaltyFLOAT0.00-2–2Penalty based on token presence. Positive values encourage new topics.
    frequency_penaltyFLOAT0.00-2–2Penalty based on token frequency. Positive values reduce repetition.
    enable_thinkingBOOLEANfalseEnable thinking/reasoning process (for Gemma, Qwen, MiniCPM, GLM, etc.).
    remove_thinkingBOOLEANfalseRemove <think>...</think> or <|channel>...<channel|> section in text
    answer_delimiterSTRINGUse this for non-standard models to clean up the output. Enter the token where the real answer starts, and the node will automatically cut out all thinking processes and technical tags generated prior to it.
    force_reasoningBOOLEANfalseFor Qwen3: force reasoning mode even on simple queries.
    words_to_banSTRINGComma-separated list of banned words. Applies logit_bias of -100 to their tokens.
    βš™οΈ Hardware & AccelerationBOOLEANtrueShow/hide group: GPU layers, MoE offload, threads, multi-GPU.
    n_gpu_layersINT-1-1–999Number of layers to offload to GPU. -1 = all, 0 = CPU only.
    n_cpu_moeINT00–128For MoE models: number of experts to keep on CPU. Saves VRAM. Slower than full GPU offload, but faster and more stable than letting the OS swap when VRAM is overcommitted.
    cpu_moeBOOLEANfalseFor MoE models: unload ALL experts into RAM. Minimal VRAM usage.
    n_threadsINT81–256Number of CPU threads to use for inference (and prompt processing).
    flash_attn_typeCOMBO-1=AUTOFlash Attention backend for llama.cpp. Requires a compatible build.
    split_modeCOMBO0=NONEGPU splitting mode: 0=NONE, 1=LAYER, 2=ROW.
    main_gpuINT00–16Index of the primary GPU when split_mode=NONE.
    cuda_deviceSTRINGSets CUDA_VISIBLE_DEVICES before init. Single index or comma-separated list.
    tensor_splitSTRINGFractions of the model to offload to each GPU (split_mode=LAYER).
    πŸ’¬ Chat, Prompts & VariablesBOOLEANfalseShow/hide group: chat handlers, formats, system prompts, and variables.
    chat_handlerCOMBOnoneChat handler for multimodal models.
    chat_formatCOMBOnoneChat format for text-only models.
    chat_format_from_ggufBOOLEANfalseForce loading chat template from the GGUF metadata. Note: Does not work with images/audio/video.
    system_prompt_defaultSTRINGDefault system prompt for the model.
    system_preset_to_user_promptBOOLEANfalseMove the system preset from the system prompt role to the user prompt role.
    user_prompt_after_contentBOOLEANtrueInsert user_prompt after the image/audio/video content. False = before.
    enable_variablesBOOLEANfalseEnable substitution of {placeholders} in system and user prompts.
    add_vision_idCOMBOautoAdd vision ID token. 'auto' = script decides (True if images != 1 or video > 0).
    add_image_idSTRINGTemplate to label images before insertion. {num} = image index (0,1,2...)
    add_frame_idSTRINGTemplate to label video frames before insertion. {video_num} = video index (0,1,2...). {frame_num} = frame index (0,1,2...)
    add_audio_idSTRINGTemplate to label audio clips before insertion. {num} = audio index (0,1,2...)
    πŸ“ Prompt TemplateBOOLEANfalseShow/hide group: custom raw prompt templates and stop sequences.
    raw_modeBOOLEANfalseEnable custom raw prompt template mode (bypasses chat handlers).
    prompt_templateSTRINGCustom prompt template. Must include {system}, {images}, {user}.
    stopSTRINGStop sequences. JSON list: ["</s>", "[INST]"] or comma-separated. Empty = use handler defaults.
    πŸ–ΌοΈ Multimodal & MediaBOOLEANfalseShow/hide group: image/audio/video limits and quality.
    force_mmprojBOOLEANtrueLoad mmproj even without media inputs (preserves template for enable_thinking).
    image_min_tokensINT00–65536Minimum tokens for image embeddings. 0 = not set.
    image_max_tokensINT00–65536Maximum tokens for image embeddings. 0 = not set.
    max_imagesINT100–100Limit on the total number of incoming images.
    max_framesINT240–512Limit on video frames. More frames require larger context.
    max_audiosINT30–100Limit on the number of incoming audio clips.
    audio_sample_rateINT00–192000Target sampling frequency for audio resampling. 0 = not set.
    image_qualityINT951–100JPEG quality (1-100) when encoding images to data URIs.
    frame_qualityINT751–100JPEG quality (1-100) when encoding video frames to data URIs (image video mode only)
    video_modeCOMBOimagesVideo processing mode. "native": Video processing on the llama side (requires llama-cpp-python version 4.0.0 or higher);uses ffmpeg for video encoding; preserves temporal sequence understanding but requires the model to support video."images": extracts frames and processes them as individual images.Use "native" for better temporal coherence, and "images" for compatibility with all models or if ffmpeg is missing from the system (not in PATH or the "python_embeded\Scripts" folder).
    video_fps_targetFLOAT1.00.1–60Target frames per second for video encoding (native video mode only). Lower values (0.5-2.0) reduce the number of frames sent to the model, saving VRAM and processing time but potentially missing fast motion. Higher values (15.0-30.0) capture more detail for fast-paced content but increase VRAM usage significantly. Typical range: 1.0-5.0 FPS for most use cases.
    video_timestamp_interval_msINT5000100–60000Interval in milliseconds between timestamp markers injected into the video stream (native video mode only). Smaller values (1000-2000ms) provide finer temporal granularity for precise event localization but increase token count. Larger values (5000-10000ms) reduce overhead for long videos where exact timing is less critical. Set to 0 to disable timestamps entirely.
    mmproj_batch_max_tokensINT10240–1048576Maximum batch size for the multimodal projector (mmproj). Multimodal tasks require more VRAM per token than standard text, so this value is typically lower than n_batch. Reduce if VRAM is insufficient (to 512 or 256) or increase for faster processing if memory allows.
    ⚑ Speculative DecodingBOOLEANfalseShow/hide group: speculative decoding parameters (requires llama-cpp-python >= 0.3.48).
    speculative_enabledBOOLEANfalseMaster switch to enable speculative decoding. Disabled automatically for multimodal inputs.
    speculative_typeCOMBO3=MTP (Multi-token Prediction)Speculative algorithm type. 3=MTP (best for Qwen3), 4/5=DFlash/DSpark (requires external draft), 7/8=N-gram (no draft model needed, good for code/JSON).
    draft_n_maxINT21–32Maximum number of draft tokens to generate per step. Recommended: 2 for MTP, 7 for DFlash.
    draft_p_minFLOAT0.000–1Minimum probability threshold to accept a draft token. 0.0 = accept all.
    draft_model_pathSTRINGPath to external draft GGUF model. Required for DFlash/DSpark. Leave empty for built-in MTP or N-gram.
    draft_n_gpu_layersINT-1-1–999Number of layers to offload for the external draft model. -1 = all, 0 = CPU.
    draft_backend_samplingBOOLEANtrueUse backend vocabulary sampler for draft tokens. Recommended True for DFlash v1 and DSpark.
    ngram_size_nINT82–32[N-gram only] Size of the n-gram window (N).
    ngram_size_mINT162–64[N-gram only] Maximum length of the draft continuation (M).
    ngram_min_hitsINT11–10[N-gram only] Minimum number of matching occurrences required to propose a draft.
    ngram_max_entries_per_keyINT41–16[N-gram K4V only] Maximum cached continuations per n-gram key.
    ctx_checkpointsINT00–4096Max number of context checkpoints per slot (0 = disabled). Set to 16 if using N-gram speculative decoding (required for rollbacks). For standard 1-question-1-answer generation or MTP/DFlash methods, keep at 0 to save memory.
    checkpoint_on_deviceBOOLEANfalseStore context checkpoints in VRAM (True) instead of RAM (False). Saves VRAM if False, but makes rollbacks slower. Only matters if 'ctx_checkpoints' (in Memory group) > 0.
    πŸ”’ Embeddings & TTSBOOLEANfalseShow/hide group: embedding/TTS extraction mode (replaces text generation).
    extract_embeddingBOOLEANfalseSwitch node to embedding mode. Uses LlamaEmbedding. Text output is replaced by a CONDITIONING tensor.
    pooling_typeCOMBO0=NONEPooling strategy for LlamaEmbedding. NONE = no pooling (per-token embeddings), MEAN = average pool, CLS = use [CLS] token, LAST = use last token.
    tokenizer_pathSTRINGPath to external HuggingFace tokenizer. Overrides built-in llama.cpp tokenizer.
    embedding_scaleFLOAT1.00.1–1000Scalar multiplier applied to the output embedding vector. 1.0 = no scaling. Useful to match the magnitude expected by downstream models.
    convert_emb_to_condBOOLEANfalseWrap the raw embedding into a ComfyUI CONDITIONING structure (hidden_states + attention_mask). Required for passing embeddings into SD/Flux conditioning slots.
    extract_ttsBOOLEANfalseSwitch node to TTS (Text-to-Speech) mode. When enabled, the node generates audio from text instead of user_prompt text. Requires mmproj_path and a TTS-compatible model.
    mmproj_use_gpuBOOLEANtrueUse GPU for mmproj (multimodal projector). Disable for CPU-only inference (slower but works without CUDA).
    mmproj_flash_attnBOOLEANtrueEnable Flash Attention for mmproj (multimodal projector). Improves performance on supported GPUs. Disable if you encounter compatibility issues.
    languageSTRINGLanguage code for TTS generation (zh, en, de, it, pt, es, ja, ko, fr, ru). Leave empty for auto-detection or model default. Note: model must support the specified language.
    πŸ› οΈ Debug, System & AdvancedBOOLEANfalseShow/hide group: logging, garbage collection, cache, and advanced settings.
    verboseBOOLEANfalseEnables verbose logging from llama.cpp.
    debugBOOLEANtrueEnables timing output for each stage to the console.
    debug_outputBOOLEANfalsePrint the final LLM text output to console.
    raw_outputBOOLEANfalseIf True, disables output.strip() (keeps leading/trailing whitespaces).
    streaming_modeBOOLEANfalseEnables token streaming to allow interrupting generation via the ComfyUI 'Interrupt' button. Adds a negligible overhead (~1%), but guarantees you can manually stop long responses. Recommended if you often need to cancel generations.
    clearing_cacheBOOLEANtrueClear cache to prevent execution freezing during heavy memory activity.
    force_gc_startBOOLEANfalseForce garbage collection after memory clearing when unload_all_models is active.
    force_gc_unloadBOOLEANfalseForce garbage collection after deleting the LLM model.
    scriptSTRINGqwen3vl_run.pyName of the Python script to execute.
    extraSTRINGJSON dict of extra keys passed to the backend script.
    config_overrideoptSTRINGStackable config override (JSON or plain text). Applied last, highest priority.

    Outputs (2)

    NameTypeDescription
    configSTRINGβ€”
    diff_configSTRINGβ€”