DoRA Extract (Knee Detection)

If you've got a base model and a full fine-tune of it, you don't have to ship the whole fine-tune - you can extract the difference as a small adapter file instead. That's LoRA extraction, and it's a well-established move (kohya's scripts have done it for years, and this pack credits them directly). This node does the same core thing but targets DoRA's format instead of plain LoRA, and it does the hardest part of the job - picking how much rank each layer actually needs - automatically, with no target percentage or ratio for you to guess at.

Some context on DoRA itself, because it matters for whether this node is the right tool: DoRA decomposes a weight update into a magnitude component and a direction component and trains both, which behaves more like a genuine fine-tune than a standard LoRA does. It had a real moment of hype in 2024 and never fully converted - the community's honest retrospective is that it doesn't stack well with other adapters and that likeness gains don't reproduce reliably across training runs. But extraction is a different job than training from scratch. You're not training a DoRA here, you're taking two checkpoints you already have and distilling their difference into a small file - and if whatever you're loading it into supports DoRA's magnitude split, this is a legitimate way to get there without running an external script.

How it works. You give it model_a (the fine-tune) and model_b (the base), and for every weight matrix it computes the difference and runs an SVD on it - the standard way to find a low-rank approximation of a weight delta. Instead of you picking a rank, this variant runs knee detection on each layer's singular-value curve: singular values drop off fast at first, then flatten out, and the "knee" is the point where that curve bends from steep to flat. Everything past the knee is contributing very little, so the node keeps only what's before it. Different layers bend at different ranks, which is the whole point - a layer barely touched by the fine-tune gets a tiny rank, a heavily modified attention layer gets a much bigger one, all without you specifying either number.

Inputs that matter. model_a and model_b - tooltips spell it out plainly: "A − B = LoRA," so A is your fine-tune, B is the base it was trained from. knee_method (default "sv_knee") selects which detection algorithm runs the curve analysis. linear_max_rank and conv_max_rank (both default 128, up to 16384) cap the rank the knee detector is allowed to land on, split separately for attention/linear layers versus convolutional ones - the same distinction LyCORIS's LoCon variant draws, since conv layers often need a different rank ceiling than linear ones to represent well.

A handful of knobs handle real architecture quirks: chunk_large_layers (off by default) splits big fused QKV/MLP blocks - common in transformer-based diffusion models - into smaller chunks before extraction, which can matter for memory. clamp_quantile (default 0.99) clips outlier singular values so a single huge one doesn't distort the rank decision. min_diff (default 0) skips layers where A and B are near-identical, saving time on parts of the model the fine-tune never touched. mismatch_mode (default "skip") handles layers present in one model but not the other.

Outputs and plumbing. output_path (STRING) is where the extracted adapter lands; it's an output node, so it runs as a terminal step regardless of downstream wiring. output_filename (default "extracted_lora"), save_dtype (default fp16), and device (default cuda) round out the basics. skip_patterns (multiline text, regex by default or glob if glob_skip_patterns is on) lets you exclude specific layers by name. lazy_load (on by default) streams tensors instead of loading the whole model at once; force_clear_cache (on by default) frees CUDA memory after each layer.

Installing it. ComfyUI Manager, search "Model Utility Toolkit," or:

cd ComfyUI/custom_nodes
git clone https://github.com/silveroxides/ComfyUI-ModelUtils

then restart.

Troubleshooting. This node needs two full checkpoints loaded at once for the diff, so expect it to be memory-hungry - lazy_load and force_clear_cache are both on by default for exactly that reason, and turning either off is a fast way to run out of VRAM on anything but a small model. If the result seems to have picked suspiciously low ranks everywhere, raise min_diff slightly and check that model_a and model_b are actually the pair you think they are - feeding it two unrelated checkpoints (rather than a base and its own fine-tune) produces a "diff" that's mostly noise, and knee detection on noise tends to land on small, useless ranks across the board.