DoRA Learned Extract (Frobenius) (WIP)

Marked "(WIP)" in its own display name, so treat everything below as a documented capability rather than a settled, battle-tested one - this is one of five "Learned Extract" nodes in the pack that add a gradient-descent refinement step on top of the usual SVD-based DoRA extraction, and it's the one that targets Frobenius norm specifically rather than a rank you set by hand.

If you've used DoRAExtractQuantile, this will look familiar - both let you set a target percentage per layer instead of a fixed rank. But they're measuring slightly different things, and it's a real distinction, not just naming. Quantile targets cumulative singular-value magnitude: keep adding singular values, largest first, until their sum hits your target fraction of the total sum. Frobenius targets cumulative squared magnitude - the Frobenius norm is, by definition, the square root of the sum of squared singular values, and it's the standard measure of how much "energy" a low-rank approximation retains relative to the original matrix (this is the same quantity the Eckart-Young theorem is about: SVD truncation is the provably optimal way to minimize the Frobenius-norm error of a low-rank approximation). In practice the two targeting methods often land on similar ranks for a given layer, but not identical ones - squaring disproportionately weights the largest singular values, so a Frobenius target tends to be a little more willing to cut a long tail of small values than a quantile target set to the same nominal percentage.

What "Learned" adds on top. Rather than stopping once SVD hits the target, this node initializes from that SVD result and then runs gradient descent to directly minimize reconstruction error against the real weight delta - the same refinement idea behind all five nodes in this sub-family, on the theory that closed-form SVD optimality per layer isn't necessarily optimal for how the extracted adapter behaves once loaded.

Inputs that matter. model_a (finetune) and model_b (base) set up the diff being extracted. linear_target and conv_target (both default 0.9) are the Frobenius-retention targets, split between linear/attention and convolutional layers. linear_max_rank and conv_max_rank (both default 128) cap how much rank a layer can claim regardless of what hitting the target would otherwise require.

The optimization controls match the rest of the Learned family: optimize_iters (default 500, 0 skips refinement entirely), optimizer (default "prodigy", with the tooltip noting AdamW wants a learning rate around 0.01 versus Prodigy's roughly 1.0), learning_rate (default 1), lr_schedule (default "plateau") with lr_patience (default 2), lr_factor (default 0.9), and lr_cooldown (default 2), plus three early-stopping conditions - early_stop_loss (default 1e-6), early_stop_stall (default 2000), early_stop_lr (default 9.01e-9) - so it doesn't burn the full iteration budget once it's converged.

Shared plumbing rounds it out: chunk_large_layers for fused QKV/MLP layers, clamp_quantile (default 0.99), min_diff (default 0) to skip barely-touched layers, mismatch_mode (default "skip"), skip_patterns/glob_skip_patterns, output_filename (default "extracted_lora"), save_dtype (default fp16), device (default cuda), lazy_load (on by default), force_clear_cache (on by default).

Output. output_path (STRING). Output node - runs as a terminal step whether or not that path is wired anywhere else.

Installing it. ComfyUI Manager, search "Model Utility Toolkit," or:

cd ComfyUI/custom_nodes
git clone https://github.com/silveroxides/ComfyUI-ModelUtils

then restart.

Troubleshooting. If you're unsure whether to reach for this over the plain DoRAExtractQuantile, the honest answer given the WIP label is: try the non-learned quantile node first, since it's the faster, more predictable path, and only reach for this one if you specifically want the gradient-descent refinement and are prepared to verify the result rather than trust it blind. Watch optimize_iters the same way you would on any Learned node here - if it consistently runs to the full budget without an early stop triggering, your learning rate is probably mismatched to whichever optimizer you've selected, and that's worth fixing before you draw any conclusion about whether the refinement itself is helping.