FP8 / MXFP8 Model Patch (SM89+)

If you already download fp8 weight files, this node looks redundant. It kind of is - and that's not the point. FP8ModelPatch doesn't load a quantized checkpoint; it takes whatever model you've already got in memory, rewrites its Linear layers to run as FP8 GEMM on the tensor cores, and hands you a patched model that drops straight into your KSampler. On a 4090 or 5090 the pitch is speed, not just the VRAM halving fp8 is famous for. The README's own chart shows a 4090 going from ~82 TFLOPS on a BF16 baseline to ~380 with torch._scaled_mm and 473 with the CUTLASS kernel this thing can use.

What it actually does

It's a MODEL-patching node in the optimization category. Wire Load Checkpoint → FP8ModelPatch → KSampler. Under the hood it clones your model (so the original object stays untouched), walks every submodule, and swaps each nn.Linear for a patched FP8Linear. Weights are quantized once at construction to float8_e4m3fn with a per-tensor scale; activations are quantized dynamically on every forward pass.

Why reach for it instead of an fp8 checkpoint? Because fp8 weight files are a fixed quantity and this is a knob. You keep your normal bf16 checkpoint, and you still get fp8's good properties: linear weights drop from 2 bytes to 1 byte per parameter, and - unlike GGUF, which has to dequantize, patch, and requantize every layer when you stack a LoRA - fp8 holds its speed. That's the exact fp8-vs-GGUF tradeoff the community settled back in the Flux era.

One honest caveat the README buries: the memory saving happens after the model is already loaded in fp16, so this won't rescue a card that can't hold the checkpoint in the first place. For fitting at all, GGUF is still the tool. For squeezing throughput out of Ada or Blackwell, this is the tool.

The mechanism

The clever part is backend routing, decided once at import time in fp8_linear.py:

• RTX 4090 (SM89) + IST-DASLab gemm-fp8 installed → the CUTLASS kernel from the 473-TFLOPS article.
• RTX 4090 without it → falls back to torch._scaled_mm, still ~4.6x faster than BF16.
• RTX 5090 (SM120) → native torch._scaled_mm with use_fast_accum=True - FP16 accumulators, the trick that work reverse-engineered for SM89.

That accumulator input is literally the fast-accumulator switch, exposed. mxfp8 mode instead uses torchao's block-scaled quantization (one scale per 32 elements instead of per tensor), which is why the README rates its accuracy better - but it's Blackwell-only, and the node hard-raises a RuntimeError if your card is SM100 or below.

The inputs that matter

• model (MODEL) - anything ComfyUI loads: Flux, SDXL, SD3, whatever.
• mode (fp8 | mxfp8) - the big decision. fp8 works on 4090 and 5090. mxfp8 is 5090 only.
• accumulator (fp16 | fp32) - fp8 mode only: fp16 is max speed, fp32 is safer numerics. Heads up: in mxfp8 mode this input does nothing at all, which is easy to miss.

Output is a single model (MODEL) that goes into your KSampler. Your GPU, mode, and chosen backend print to the console on startup.

Install

ComfyUI Manager, search fp8_cublas, install - or:

cd ComfyUI/custom_nodes
git clone https://github.com/marduk191/ComfyUI-fp8_cublas

Then torchao>=0.10.0 for MXFP8 (bundled with recent ComfyUI; pip install torchao if not). The optional-but-recommended gemm-fp8 for 4090s is the fiddly one - it has no PyPI release, so no requirements file can install it:

git clone --recurse-submodules https://github.com/IST-DASLab/gemm-fp8
cd gemm-fp8 && pip install -e .

Skip that and you still get the torch._scaled_mm path, just with a warning at startup.

Troubleshooting & gotchas

• RTX 3090 or older: both modes raise. This is an SM89+/SM100+ toy. On a 30-series, INT8-ConvRot or GGUF is your lane instead.
• 4090 + mxfp8: RuntimeError. Needs SM100+. The recommended 4090 config is fp8 / fp16.
• gemm-fp8 won't compile? The fallback is automatic; don't let it block you.
• It only patches nn.Linear layers with both dimensions ≥ 64. Flux's DiT is almost all Linears, so the win is big; SDXL's UNet leans on 1x1 convs, so the win there is smaller. Temper expectations per model.
• It's an inference patch - don't train on it.

One nice combo the README calls out: safe to stack with SageAttention, since they patch different ops. On a 5090: FP8ModelPatch (mode mxfp8) → Patch Sage Attention → KSampler.

Bottom line: if your card is below a 4090, walk away. If you just need VRAM headroom, an actual fp8 checkpoint is more direct. But if you own a 4090 or 5090 and your sampling is compute-bound, this is a several-times speedup on matmuls you're already running, for a three-minute install.