Quantization Config Node

Local LLMs are thirsty. A model that's fine in fp16 on a server card eats your whole 12GB consumer GPU at load. The Quantization Config Node is this pack's answer: it builds a BitsAndBytesConfig for the transformers library so the main LLM_Node loads models in 8-bit or 4-bit instead of full precision. Roughly speaking, 4-bit is a ~4x memory cut versus fp16 for modest quality loss - the difference between "won't fit" and "runs fine."

What it is

It's a config node: no text in, one QuantizationConfig out. Every input maps 1:1 onto a BitsAndBytesConfig keyword, and the pack's source passes it straight into from_pretrained as the quantization_config. The list of knobs is intimidating, but you really only need one:

• quantization_mode - none, load_in_8bit, or load_in_4bit. This is the switch that matters. Default none.

The rest are the fine print, all safe to leave at defaults: llm_int8_threshold, llm_int8_skip_modules, llm_int8_enable_fp32_cpu_offload, llm_int8_has_fp16_weight, and the 4-bit set - bnb_4bit_compute_dtype, bnb_4bit_quant_type (fp4 vs nf4), bnb_4bit_use_double_quant, bnb_4bit_quant_storage. If you ever graduate past flipping the mode, the two worth touching are bnb_4bit_quant_type (nf4 is generally the quality pick) and bnb_4bit_use_double_quant (shaves a bit more memory).

What actually applies

Two important caveats, both grounded in the code. First, this node only affects the transformers model path. If your model folder has "GGUF" in the name it goes through llama.cpp instead, and GGUF files are already quantized - the quantization config is simply never consulted there. For genuinely big models you now have two legitimately good routes: this node's bitsandbytes 4-bit, or a GGUF file at Q4-K_M. Both are fine; GGUF skips the bitsandbytes dependency entirely, which is a real advantage on setups where that library is fussy.

Second, none still constructs an empty BitsAndBytesConfig (both load flags false) and passes it along - harmless in practice, but it means bitsandbytes needs to be importable even when you aren't quantizing, which is the setup gotcha below.

Installing and the bitsandbytes requirement

Pack install is the usual: ComfyUI Manager, searching "LLM Node", or

cd ComfyUI/custom_nodes
git clone https://github.com/Big-Idea-Technology/ComfyUI_LLM_Node

then restart. But note the split: requirements.txt lists transformers, torch, accelerate, and llama-cpp-python - bitsandbytes is not in it. The pack's install.sh installs it separately (pip install -i https://pypi.org/simple/ bitsandbytes). Manager usually runs that install script for you, but if you cloned manually or bitsandbytes is missing, install it into ComfyUI's Python environment yourself. If you get an import error the moment quantization is involved, that's your culprit. And if bitsandbytes won't cooperate on your platform, remember the GGUF route in the same pack needs none of it.