H3 Fun ControlNet Apply

MiniMax-H3 is great at turning a prompt and a first frame into a clip. What it's not good at is hitting a specific action - a figure walking laterally instead of drifting toward the lens. H3FunControlApply fixes that. It's the first working ComfyUI implementation of the H3 Fun-ControlNet: feed it a control video - depth, canny, pose, HED or MLSD - and the generation follows that structure frame by frame.

Alibaba PAI released the Fun-ControlNet weights, Kijai re-derived them for the pruned checkpoints, and nobody had wired them into ComfyUI - until now. Where the control video comes from is your call: a ComfyUI preprocessor, a depth pass out of a 3D package, anything. It just needs an IMAGE batch.

How it works

It takes control_video, VAE-encodes it once (not per sampling step), packs the latent into 196 columns - control latent, mask, masked latent - and injects it at base layers 0/10/20/30/40 of H3's 50. The tower is built from ComfyUI's own H3 modules, riding the patches_replace["dit"] extension point.

It's a MODEL in, MODEL out node - it patches the model, never the conditioning - so your conditioning can come from any H3 node:

UNETLoader --> H3FunControlApply --> MiniMaxH3SigmaShift --> Guider --> Sampler
                     ^ control video                          ^
               H3FunControlLoader                  any H3 conditioning node

The skip lands on the video rows only, zero-gated - strength 0.0 is a true bypass, and the audio track survives. H3 packs video and audio into one token stream, and a naive earlier port silently corrupted the sound.

The inputs that matter

• control_video - must match your generation's length, width and height exactly. A mismatch is a hard RuntimeError naming both token counts.
• strength - scales every skip, and it's a shared budget when you chain: the skips sum, so depth 1.0 plus pose 1.0 behaves like a single tower at 2.0 and saturates - washed out, figure smeared. The author's tested pairing is depth 0.3 + pose 0.7. Don't reach for 1.6 to fix weak adherence; it just degrades the picture.
• start_percent / end_percent - the slice of sampling where control applies. Not cosmetic. Defaults run control through the final texture-forming steps, which can fight the prompt on featureless regions - a flat ground plane beats the control every time. Ending around 0.6 pins structure while noise is coarse and lets late steps add detail.

One finding before you blame the node: with depth alone the figure drifts in distance and grows across the shot; pose holds it steady. Token resolution is why - H3's VAE downsamples 16x, so a small subject occupies almost no depth ramp. Frame larger or generate higher; that's a shot decision, not a node setting.

Install

Same pack as the loader, no dependencies beyond ComfyUI (it's built on comfy.ldm.minimax; needs ComfyUI ≥ 0.34.0, CUDA):

cd ComfyUI/custom_nodes
git clone https://github.com/wyzborrero/ComfyUI-H3-FunControl

Or find ComfyUI-H3-FunControl in Manager, then restart. You also need the curve-form checkpoint - minimax_h3_fun_controlnet_union_pruned_bf16.safetensors from Kijai's MiniMax-H3-experimental - in models/controlnet, and the vae input takes the H3 video VAE. The bundled 01_single_control workflow is a good starting point.

Where people get burned

This README is basically a list of ways the pipeline fails silently - plausible video, wrong result, no error. Two will bite you:

• A node edit is inert until restart. ComfyUI imports custom nodes once at startup. The author once chained two applies and got output bit-identical to pose-alone - the second apply had silently overwritten the first. If "nearly identical" could mean "one branch never ran," change the seed and diff.
• Sparse-attention speedups reorder tokens. If you run Sol-Attn, keep morton off - the control skip lands at raster-order row offsets, so Z-order permutation drops it entirely, static figure, no error. With morton: false it's ~1.5x with control intact.

For characters, pair this with MiniMaxH3ReferenceToVideo rather than MiniMaxH3ImageToVideo - the latter only takes first_frame, so an untextured 3D render bakes in untextured. A reference image on every frame plus the ControlNet driving motion is the difference between a nude mannequin and the actual character.

Two caveats. CFG: the default is "guidance 1, it's baked in," but the author measured 4.0 markedly better for a detailed style brief at 28 steps - and BasicGuider has no negative input, so negatives are inert until you switch to CFGGuider. And the ControlNet was trained on fl2va while most run ref2va: it loads and runs, adherence appears to hold, but it's not the trained pairing. If adherence is weird, that's the first thing to question.