H3 Streamed Blocks (exact low-VRAM, alpha)
The de-rope pass on a 16 GB card, bit-equal to the 96 GB one
- model
- MODEL
H3 Streamed Blocks is the low-VRAM answer to the de-rope's least friendly habit: a long dilated pass on MiniMax-H3 materializes its fused QKV and SwiGLU tensors for the whole sequence at once. At 217k packed tokens that's 8.6 and 15.4 GiB respectively - enough to OOM a 24 GB card and, at 1376x768, even a 96 GB one. This node runs every DiT block in token chunks instead, with the same math, and the claim is the good kind of audacious: bit-equal to stock, because the int8/W4A8 activation quantization is per-row and accumulates in int32. Measured on same-seed renders, video and audio both.
How it works
It patches your model, so model is the only input you can't avoid. The dials are chunk sizes, and the defaults (16384) are sane:
q_chunk- query tokens per attention call. Smaller = lower transient VRAM, more calls. Exact at any value, which is the design guarantee.kv_chunk- tokens per K/V projection chunk while building the full K/V buffers.mlp_chunk- tokens per MLP chunk; 0 leaves the model'smlp.forwardalone (e.g. for KJNodes' own chunk node).min_tokens- below this packed sequence length (default 32768) the stock block runs unchanged, because short clips gain nothing.
The optional kv_store is where the interesting engineering lives, and the honest labels are in the enum: bf16 (exact) is the bit-equal default; kvi8r keeps K/V as rotated int8 (halves the bytes, approximate - a "sibling take" the operator judged almost perfect); kvi8s uses SageAttention's layout attended on int8/fp8 tensor cores with no dequant (needs the sageattention package, 2.x); kvfp4s goes one rung further on Blackwell fp4 kernels (fastest, most approximate, needs SageAttention3's fp4 extension); kvmix blends 4- and 8-bit per head. final_layer_chunk streams the output head (stock promotes the whole span to fp32 twice - ~10 GiB at 216k tokens, the forward's peak), and trim_forward releases the patch-embed buffers once the packed sequence is assembled. self_check is a diagnostic that runs stock and streamed side by side on block 0 and logs the divergence - costs one extra block forward, worth doing once on your own card.
The real-world shape of the win
The README's headline: with ComfyUI's dynamic VRAM and --fast-disk, the 702-frame pass that OOMed a 96 GB card renders on a 16 GB card in a 32 GB machine at essentially the same seconds per step (316 vs 311), because at that length the step is attention-bound and the weight traffic hides underneath. It needs a recent ComfyUI (developed on 0.33.0, imports comfy.ldm.minimax.model and PyTorch flash attention). It's loaded behind the pack's guarded loader, so if the module can't import, the rest of MAINodes still loads and the console prints one line telling you why - paste that line into an issue.
Install
Ships with matlowai/ComfyUI-MAINodes:
cd ComfyUI/custom_nodes
git clone https://github.com/matlowai/ComfyUI-MAINodes
Restart, under MAINodes/VRAM Lab. No extra Python deps beyond the optional sageattention/sageattention3 for the approximate kv_store rungs; you need MiniMax-H3 weights and ComfyUI's H3 support (mind the licence's US/EU/UK/Korea carve-outs).
Two things to remember: this node changes nothing unless it's in the graph, and the bf16 (exact) path is the one to trust while you're comparing takes. The approximate rungs are faster and cheaper but they're "sibling takes" by the author's own accounting - flip to them once you've locked the look, not while you're judging it.
Inputs (12)
| Name | Type | Default | Description |
|---|---|---|---|
| model | MODEL | — | |
| q_chunk | INT | 163841024–262144 | Query tokens per attention call. Smaller = lower transient VRAM, more calls. Exact at any value. |
| kv_chunk | INT | 163841024–262144 | Tokens per K/V projection chunk while building the full K/V buffers. |
| mlp_chunk | INT | 163840–262144 | Tokens per MLP chunk. 0 = leave the model's mlp.forward alone (e.g. KJNodes' chunk node). |
| min_tokens | INT | 327680–1048576 | Below this packed sequence length the stock block runs (short clips gain nothing). |
| kv_block | INT | 00–262144 | EXPERIMENTAL, leave at 0. Attends K/V in blocks with a log-sum-exp combine (1 bf16 ulp). As built it does not lower memory (K/V are still fully built; measured +1.1 GiB and ~7% slower at 216k tokens); kept for the host-staged K/V design to come. |
| final_layer_chunkopt | INT | 163840–262144 | Rows per chunk through the output head's norm -> mod -> fp32 promotion. Stock promotes the whole span to fp32 twice (~10 GiB at 216k tokens, the forward's peak). 0 = stock. |
| final_layer_gemmopt | COMBO | exact (whole GEMM, one fp32 buffer) | exact: same head GEMM as stock, transient = one fp32 [rows, hidden] (bit-equal). streamed: GEMM per chunk, transient ~chunk-sized, but fp32 cuBLAS is not chunk-invariant (numerically-equivalent tier). |
| kv_storeopt | COMBO | bf16 (exact) | kvi8r = rotated int8 K/V. K and V held as int8 with one fp16 scale per row (per token, per head) in a fixed orthonormal 128-wide Hadamard-rotated basis of the head dim; the query chunk is rotated to match and the output un-rotated once, so a K/V block dequant is a single int8->fp16 cast * scale and attention runs in fp16 blockwise (kv_block, default 16384) with an online-softmax combine. Halves the K/V bytes. NOT bit-equal to stock: a same-seed render is a sibling take (first cut 2026-08-18: operator judged the de-rope side by side 'almost perfect'). Second cut (2026-08-18 evening): standalone at 217k tokens the attention costs +16% over the exact path (first cut +24%) with a ~1 GiB transient (first cut ~3), so the K/V saving now shows in the forward peak; the live numbers are in LOWVRAM.md. kvi8s = the same K/V bytes kept in SageAttention's kernel layout (int8 K per 64-token block, fp8 V per channel, Q/K Hadamard-rotated first) and attended on int8/fp8 tensor cores straight from the store, no dequant: standalone ~1.6x faster attention than the exact path at 217k tokens, one rung more approximate than kvi8r; needs the sageattention package (2.x). kvfp4s = the same idea one precision rung further down, on SageAttention3's Blackwell fp4 kernels: K/V kept as NVFP4 (E2M1 + one e4m3 scale per 16 elements = 0.5625 byte/elem, 0.28x bf16 -- 1.63 GiB vs 5.79 at 216k tokens), Q/K Hadamard-rotated, no delta_s and no Q-centering (measured to cost nothing at per_block_mean False, which is the only affordable setting at H3 lengths). Standalone at 217k tokens: ~2.4x faster attention than the exact path and ~1.4x faster than kvi8s, at rel-rms 0.293 vs Sage 2.2's 0.055 on the outlier proxy (0.192 vs 0.039 plain gaussian) -- clearly the most approximate rung, gated on the operator's eyes. Because the fp4 kernel's log-sum-exp is not retrievable there is no K-blocked combine: phase 1 buffers K/V in bf16 and quantises once, so the block's peak during that one finalize is bf16 + fp4 before the bf16 is dropped. Needs SageAttention3's fp4 extension (fp4attn_cuda) importable; falls back to bf16 (exact) with a warning if it is not. |
| trim_forwardopt | BOOLEAN | true | F5: run a copy of the stock model forward that releases the patch-embed and row buffers once the packed sequence is assembled (stock keeps them for the whole forward: 2.14 GiB at 216k tokens). Same math, exact. Applied only if the installed ComfyUI's forward matches the copy (source hash); otherwise skipped with a log line. |
| self_checkopt | BOOLEAN | false | Diagnostic: on block 0's first call, run stock and streamed on the same input and log per-phase divergence. Costs one extra block forward. |
| exact_av_rowsopt | BOOLEAN | false | kvfp4s only. Route the TEXT and AUDIO query rows through an exact bf16 attention over a retained bf16 K/V, leaving the video rows (96.3% of the packed sequence, where the speed win lives) on the fp4 store. Audio is ~300 of 28,931 tokens, so the ~4% attention error that video's redundancy hides as slight ghosting lands audibly on a soundtrack carried by 1% of the sequence -- measured: audio is not quantised worse than video, it is merely far more sensitive, so the fix is routing rather than a better quantiser. Costs the bf16 K/V of the block in flight (~0.8 GiB at 29k tokens, 5.8 at 216k) and ~5% more attention time. No effect unless kv_store is kvfp4s. |
Outputs (1)
| Name | Type | Description |
|---|---|---|
| MODEL | MODEL | — |