TE DoRA Extract (Frobenius)
Extract a text-encoder DoRA that keeps a target share of the energy
- output_path
The fixed-rank extractor from this pack makes every layer the same size whether it needs it or not. This one does the opposite: you tell it how much of each layer's "energy" to keep, and it lets the rank vary per layer to hit that target. The target is expressed as a fraction of the Frobenius norm - the standard measure of how much a matrix's magnitude lives in its largest singular values. It's the "I want this DoRA to be as small as it can be while keeping 90% of what made the fine-tune different" mode.
How it works
Model A is the fine-tuned text encoder, Model B is the base. Per layer, the node computes A − B, SVD-decomposes the delta, and picks the rank at which the retained singular values cover linear_target (for linear/attention layers) or conv_target (for conv layers) of the total Frobenius norm. Layers with a rich, spread-out spectrum get a higher rank; layers whose delta is concentrated in a few singular values get a low one. Either way, the DoRA structure - low-rank down/up factors plus the dora_scale magnitude vector - is preserved, and the result is written to your LoRA directory.
Two controls keep the per-layer rank sane. probe_offset (default 32) samples extra singular values beyond the max rank so the rank decision isn't made at the edge of a truncated spectrum, and linear_max_rank / conv_max_rank (defaults 128) cap how large any layer can get. So the effective rule is: keep target fraction of the energy, but never above the max rank, and probe a bit past it to make the decision trustworthy.
Inputs that matter
- model_a / model_b - fine-tuned and base text encoders (
A - B = LoRA), both frommodels/text_encoders. - linear_target / conv_target (defaults 0.9) - the Frobenius-norm fraction to retain per layer type. 0.9 is the classic choice; 0.95 buys a little more fidelity at more size, 0.8 shrinks aggressively.
- probe_offset (default 32) - extra singular values sampled beyond max rank for a reliable decision.
- linear_max_rank / conv_max_rank (defaults 128) - hard caps per layer type.
- output_filename (default
extracted_te_lora) - written to the LoRA directory. - save_dtype (default fp16), clamp_quantile (0.99), min_diff (0), chunk_large_layers (off) - same meanings as the fixed-rank sibling.
- mismatch_mode, skip_patterns / glob_skip_patterns, lazy_load (on), force_clear_cache (off), device - the usual handling.
Outputs
output_path - a terminal node; the file it writes is the deliverable.
When to pick Frobenius
When you don't know the right rank and want the extraction to spend its budget where the fine-tune actually moved. It's the middle path between fixed rank (predictable, wasteful) and knee detection (fully automatic, occasionally fiddly) - you control the quality floor and the size ceiling, and the node does the per-layer arithmetic. For a first extraction from an unfamiliar pair of text encoders, this is the one to try.
Install
Part of Model Utility Toolkit (silveroxides/ComfyUI-ModelUtils). ComfyUI Manager → search "Model Utility Toolkit", or:
cd ComfyUI/custom_nodes
git clone https://github.com/silveroxides/ComfyUI-ModelUtils
Restart ComfyUI. Real dependency: unifiedefficientloader (UEL). Keep ComfyUI current - the pack uses the newer extension API. Extraction lineage is kohya-ss/sd-scripts + LyCORIS, per the pack's acknowledgements.
Inputs (18)
| Name | Type | Default | Description |
|---|---|---|---|
| model_a | COMBO | Finetuned Text Encoder model (A - B = LoRA) | |
| model_b | COMBO | Base Text Encoder model (A - B = LoRA) | |
| linear_target | FLOAT | 0.900–1 | Target Frobenius-norm fraction retained for linear layers. |
| conv_target | FLOAT | 0.900–1 | Target Frobenius-norm fraction retained for convolution layers. |
| probe_offset | INT | 321–4096 | Extra singular values sampled beyond Max Rank for a reliable bounded rank decision. |
| linear_max_rank | INT | 1281–16384 | Maximum extracted rank for linear layers. |
| conv_max_rank | INT | 1281–16384 | Maximum extracted rank for convolution layers. |
| lazy_load | BOOLEAN | true | Low memory mode: load tensors from disk on demand |
| force_clear_cache | BOOLEAN | false | Clear CUDA cache after each layer; slower but useful under severe VRAM pressure. |
| chunk_large_layers | BOOLEAN | false | Split large fused layers (QKV, MLP) into chunks |
| clamp_quantile | FLOAT | 0.990.5–1 | Clamp outlier singular values |
| min_diff | FLOAT | 0.0000–1 | Skip layers with max difference below this |
| mismatch_mode | COMBO | skip | Handle missing or incompatible text-encoder tensors by skipping them, substituting zeros where supported, or aborting. |
| output_filename | STRING | extracted_te_lora | Output filename without extension, written under the ComfyUI LoRA directory. |
| save_dtype | COMBO | fp16 | Data type used to save extracted text-encoder LoRA factors. |
| device | COMBO | cuda | Device used for per-layer extraction arithmetic; CUDA out-of-memory retries the affected layer on CPU where supported. |
| skip_patterns | STRING | Patterns for layers to skip (regex or glob depending on glob_skip_patterns) | |
| glob_skip_patterns | BOOLEAN | false | When True, skip_patterns use glob syntax (* = any sequence, ? = any char, dots are literal). When False (default), patterns are Python regex matched as substrings. |
Outputs (1)
| Name | Type | Description |
|---|---|---|
| output_path | * | — |