DoRA Learned Extract (Knee Detection) (WIP)

This is DoRAExtractKnee with a gradient-descent stage bolted on, and it carries the "(WIP)" tag in its own display name to prove it - the pack's author is telling you plainly that this is the newest, least-battle-tested corner of the extraction tools, not a settled feature. If you want the hands-off version, automatic rank selection with no learned refinement, use DoRAExtractKnee instead; this one is for when you want the same automatic per-layer rank decision but are willing to spend extra compute trying to squeeze out a better fit.

The two-stage idea is the same across all five "Learned" nodes in this pack: run knee detection on each layer's singular-value curve to find the natural elbow - the point where the curve stops dropping steeply and flattens out, meaning everything past it contributes little - use that as your rank and your SVD-based starting point, then run gradient descent to directly minimize the reconstruction error between the extracted low-rank adapter and the actual weight delta. Plain SVD is provably optimal for a low-rank matrix approximation in isolation, but that guarantee doesn't account for how DoRA's magnitude-plus-direction split behaves once loaded, or for interactions across layers - refining past the SVD answer is a bet that direct loss minimization can do better on the thing you actually care about.

Inputs that matter. model_a (finetune) and model_b (base) set up the extraction. knee_method (default "sv_knee") picks the curve-detection algorithm - same field as the non-learned DoRAExtractKnee. linear_max_rank and conv_max_rank (both default 128) still cap whatever rank the knee detector lands on, split for linear/attention versus convolutional layers.

The optimization stage brings the full Learned-family toolkit: optimize_iters (default 500, 0 disables refinement entirely and falls back to plain knee-based SVD), optimizer (default "prodigy" - adaptive, auto-tunes its own learning rate, which is the community's usual reason to reach for it), learning_rate (default 1, with the tooltip warning AdamW wants roughly 0.01 while Prodigy wants roughly 1.0 - don't carry one value across a switch between the two), lr_schedule (default "plateau") governed by lr_patience (default 2), lr_factor (default 0.9), and lr_cooldown (default 2), plus three early-stop conditions - early_stop_loss (1e-6), early_stop_stall (2000 iterations), early_stop_lr (9.01e-9) - so it isn't burning compute after convergence.

Shared extraction plumbing: chunk_large_layers for splitting fused QKV/MLP blocks common in transformer diffusion architectures, clamp_quantile (default 0.99) to bound outlier singular values, min_diff (default 0) to skip layers the fine-tune barely changed, mismatch_mode (default "skip"), skip_patterns/glob_skip_patterns for manual exclusions, output_filename (default "extracted_lora"), save_dtype (default fp16), device (default cuda), lazy_load (on by default), force_clear_cache (on by default).

Output. output_path (STRING) - an output node, so it executes as a terminal step regardless of whether anything reads that path.

Installing it. ComfyUI Manager, search "Model Utility Toolkit," or:

cd ComfyUI/custom_nodes
git clone https://github.com/silveroxides/ComfyUI-ModelUtils

then restart.

Troubleshooting. Because knee detection already makes the rank decision for you, the main thing worth testing here is whether the added optimize_iters refinement is actually earning its compute cost over the plain DoRAExtractKnee - run both against the same model pair and compare file size and, more importantly, actual behavior when loaded, rather than assuming more compute means a better result. If refinement runs to the full iteration budget without an early stop firing, that's usually a learning-rate mismatch with whichever optimizer you've picked, not a sign the model pair is unusually hard to fit. As with every extraction node in this pack, two full checkpoints resident at once is a real VRAM cost - keep lazy_load and force_clear_cache on unless you know you have headroom to spare.