ROCm GGUF Loader

GGUF is how most people actually run big models on small cards - a single-file quantized format where a Q4_K_M Wan 14B fits on a 10GB card and Q8 is visually indistinguishable from fp16 at half the size. The catch has always been the plumbing: GGUF files aren't PyTorch pickles, they need a loader that understands the quantized tensor layout, and for years that meant city96's ComfyUI-GGUF. This node is the ROCm Ninodes pack's own GGUF loader - a self-contained implementation that scans your model folders for .gguf files, loads them, and handles the dequantization internally.

What it does

The interface is almost insultingly simple: one dropdown (gguf_name - it scans diffusion_models, unet, unet_gguf, and checkpoints folders for .gguf files), and one MODEL output that wires straight into your sampler. That's it. No precision toggles, no manual dequant knobs.

Under the hood it reimplements the GGUF loading path - reading the format spec's quantized tensor types, building GGML-style layers, and lazily converting quantized tensors to PyTorch. The source is honest about it: this is a reimplementation inspired by City96's structure, and it's a minimal-but-working path. Q4/Q5/Q8 and friends are the point; the loader detects the quant type and dequantizes on the fly. If you've got gguf>=0.7.0 in your environment (the pack pulls it in as a dependency), you're set.

The reality check

Where you'll actually use this: WAN 2.2 video (the README literally cites wan2.2_i2v_low_noise_14B_Q4_K_M.gguf as the example), Flux, and other DiT-era models that ship as GGUF packs. Video is the sweet spot - those 14B+ video models are exactly the ones that won't fit at fp16 on consumer cards, and GGUF is the community's answer.

Two honest caveats. First, on a 16GB+ card where fp8 fits, fp8 is usually the better tool - the KB's settled consensus is that fp8 is simpler, slightly faster, and holds its speed when you stack LoRAs, while GGUF pays a dequantization tax. Second, LoRAs on top of GGUF are the weak spot everywhere (including city96's own pack): each layer gets dequantized, patched, then requantized, and it's slow. If you're LoRA-heavy and capped on VRAM, the standard advice is to drop a quant level to make room rather than fight the dequant cycle. This loader doesn't change that math.

Also worth knowing: this is a diffusion model loader. It outputs MODEL only. Your GGUF text encoder (quantized T5 or Qwen3) still loads through your usual CLIP loader, and the VAE loads normally - GGUF here is for the big denoising weights.

Install and troubleshooting

Part of the ROCm Ninodes pack - Manager → "ROCm Ninodes," or

cd ComfyUI/custom_nodes
git clone https://github.com/iGavroche/rocm-ninodes.git

Restart, find it under ROCm Ninodes → Loaders. Dependencies: numpy, Pillow, psutil, gguf>=0.7.0, safetensors. The gguf package is the important one - if the dropdown is empty or load fails with an import error, that's your pip install gguf moment.

If loading fails on a file you know is valid, the pack's docs offer the pragmatic fallback: convert the GGUF to .safetensors and use the Diffusion Loader instead. And if you're on a Strix Halo APU specifically, remember the README's launch flags (TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1) - the loader is tuned for that unified-memory setup, but it'll run on any working ROCm stack. It's a niche node in a niche pack, but for AMD users running quantized Wan or Flux, it does the job with fewer moving parts than most.