Nodes/FunUI/FunUI Train LoRA
ComfyUI Node

FunUI Train LoRA

Train on the clips you rated, not a folder you assembled

By digital-garbage·Created about a year ago·Updated about 11 hours ago· 22
FunUI Train LoRA
  • clip
  • vae
  • model
  • audio_vae
  • lora
◄set_name►
◄namefunui_lora►
◄usenormal►
◄steps500►
◄learning_rate0►
◄rank16►
◄algorithmLoRA►
◄optimizerAdamW►
◄lossMSE►
◄batch_size1►
◄grad_accumulation1►
◄resolution512►
◄aspect_bucketsfalse►
◄frames33►
◄fps0.0►
◄startstart►
◄trigger►
◄seed0►
◄training_dtypebf16►
◄lora_dtypebf16►
◄gradient_checkpointingtrue►
◄offloadingfalse►
◄bypassfalse►
◄model_name►
◄weight_dtypedefault►
◄compute_dtypedefault►
◄postfix►
◄swap_blocks0►
◄preview_every0►
◄save_every0►
◄resume►

Almost every LoRA trainer assumes you're going to spend an afternoon curating a dataset folder: 20–40 images, captioned, bucketed, uploaded to a rented GPU, watched through a loss curve. FunUI's LoRA training add-on takes the opposite angle - you already rated a few hundred of your own generations in the studio, and the ratings are the dataset.

FunUI Train LoRA is the node that does the actual training. It's the least "just try it" node in the pack: it's an add-on you switch on, it only has real mileage on one model, and the README says as much.

What it is

Train a LoRA from a saved training set on ComfyUI's own trainer, and write it into your models/loras folder. The training loop follows musubi-tuner's MiniMax H3 recipe - which layers get a LoRA, per-block checkpointing, noise sampling, loss weighting, optimizer and learning rate - and its block-swap trick, where a number of transformer blocks keep their weights in system memory.

Read that last part as the honest framing: this is not a novel trainer, it's a well-chosen existing recipe with the UI and dataset plumbing wrapped around it. That's usually the right call.

Before the node: the set

set_name is a set you built in the app's LoRA training screen - pick the data, describe each item, done. There's no way to point this node at an arbitrary folder of images; the set is the unit. And use decides what comes out of it:

  • normal - liked results plus your own media. The usual one.
  • avoid - only the disliked results, for a LoRA you then apply at negative strength. That's the surviving version of the "negative embedding" idea: a LoRA trained on bad output, used negatively. It fails loudly instead of silently, which is more than EasyNegative can say on a modern encoder.
  • all - everything in the set.

The inputs that matter

clip and vae are required because something has to turn captions into conditioning and pictures into latents. For an audio-video model, wire audio_vae too or the soundtrack isn't part of what's learned.

For the model itself you have two routes: wire a loaded model, or leave it empty and give model_name - the trainer then loads the file itself, once, with no loader node in front. The second is the one to use when VRAM is tight, because it doesn't leave a second copy of a 33B model hanging around.

The rest is a normal trainer's dials, and the pack's defaults are sane: 500 steps, learning rate 1e-4, rank 16, AdamW, MSE, batch 1, bf16 training and LoRA dtype, gradient checkpointing on. On a video model batch size is 1 by design - H3 trains one sample at a time - and grad_accumulation is how you fake a bigger batch without the memory.

Worth knowing about, because they're what makes a long run survivable:

  • save_every - dumps <name>_step<N> into the loras folder as it goes. Set it. The final epoch is not reliably the best checkpoint, and this is the cheapest insurance in the node.
  • preview_every - pauses every N steps so the pipeline can make a preview with the LoRA so far.
  • resume - the name of a paused run to pick up.
  • swap_blocks / offloading - move weights to system memory, slower, for cards that can't hold the model.
  • trigger and postfix - text wrapped around every caption, so you have a word to call the LoRA up with later.

Output is a single lora string - the file it wrote. It is not a MODEL, so it doesn't plug into a sampler. You load it with a LoRA loader afterwards, like any other LoRA. And the filename is never overwritten: a second run with the same name doesn't clobber the first.

Installing it

cd ComfyUI/custom_nodes
git clone https://github.com/digital-garbage/FunUI
pip install -r requirements.txt

Then enable it: Settings ▸ Modules ▸ LoRA training. Without that switch the node throws "LoRA training is switched off" the moment you queue. It rides on your existing PyTorch - the pack's other add-ons (spaCy for automatic camera and focus, nvidia-vfx for RTX Super Resolution) are the ones that install extra packages on the next start.

Where people get burned

Enthusiasm vs. reality. The README is explicit that the add-on has been run on MiniMax H3 only. Stable Diffusion 1.5 is marked experimental and LTX 2.x is marked untested across the whole pack. Don't be the first person to discover that training a LoRA on an untested architecture produces eleven hours of nothing.

Expecting an SDXL-style recipe to carry over. It doesn't. Rank, LR and step counts are architecture-specific; the KB's training doc has per-architecture numbers, and the honest summary is "start from the trainer's default". For H3, 16 / 1e-4 / 500 is a starting point, not a law.

A set with nothing in it for the mode you picked. Choosing avoid on a set with no disliked clips gets you a validation error naming the set, which is at least clear. Rate some clips you hated first.

CategoryFunUI/Training

Inputs (35)

NameTypeDefaultDescription
clipCLIPThe text encoder that turns the captions into conditioning.
vaeVAEThe VAE that turns pictures and clips into latents.
set_nameSTRINGThe training set to learn from.
nameSTRINGfunui_loraFile name of the finished LoRA (no extension). An earlier file is never overwritten.
useCOMBOnormalnormal: liked results and your own media. avoid: only the disliked results, for a LoRA you apply at negative strength. all: everything.
stepsINT5001–100000How many training steps.
learning_rateFLOAT01e-7–1How big each learning step is. Lower is safer and slower.
rankINT161–128How much the LoRA can learn. Higher holds more detail and makes a bigger file.
algorithmCOMBOLoRAThe adapter kind. LoRA is the common one.
optimizerCOMBOAdamWHow the weights are updated.
lossCOMBOMSEHow the error is measured.
batch_sizeINT11–64Samples per step. Models that train one sample at a time (H3) lower it to 1.
grad_accumulationINT11–256Steps added up before the weights change: a bigger batch for less memory.
resolutionINT51264–2048Training size in pixels per side. Each item is fitted to about this area, keeping its shape.
aspect_bucketsBOOLEANfalseKeep each picture's own shape (a few size groups) instead of one shape for all. Not used for H3.
framesINT331–1025Frames taken from each clip, for video models. H3 rounds to its own grid (17 x n + 5).
fpsFLOAT0.00–120Frames per second to read the clips at. 0 uses each clip's own rate (H3: 24).
startCOMBOstartWhere in a clip to take the frames from.
triggerSTRINGThe prefix: text put at the start of every caption, for example a word to call the LoRA up later.
seedINT00–18446744073709550000Seed for the noise and the LoRA's starting weights.
training_dtypeCOMBObf16Precision used while training. bf16 is right for nearly everything.
lora_dtypeCOMBObf16Precision the LoRA is stored in.
gradient_checkpointingBOOLEANtrueTrades speed for a lot less memory.
offloadingBOOLEANfalseKeeps model weights in system memory while training: slower, for cards that cannot hold the model.
bypassBOOLEANfalseApply the LoRA by hooks instead of changing weights. For quantized models.
model_nameSTRINGThe diffusion model file to train on, as the loader lists it. The trainer loads it itself, once, with no loader node in front.
weight_dtypeCOMBOdefaultHow the model file's weights are stored when it is loaded here.
compute_dtypeCOMBOdefaultWhat the maths runs in when the model is loaded here.
postfixSTRINGText put at the end of every caption.
swap_blocksINT00–128How many transformer blocks keep their weights in system memory, fetched one layer at a time: slower, and each one frees about a block's weight on the card. 0 keeps the whole model on the card.
preview_everyINT00–100000Pause every this many steps so the pipeline can make a preview with the LoRA so far. 0 never pauses.
save_everyINT00–100000Save a copy of the LoRA so far to the loras folder every this many steps, named <name>_step<N>. 0 saves only the finished one.
resumeSTRINGThe name of a paused run to carry on. Empty starts a new run.
modeloptMODELA model already loaded to train on. Leave empty and give model_name to have the trainer load the file itself, once.
audio_vaeoptVAEOnly for models that learn sound with the picture (H3): turns the clips' sound into latents.

Outputs (1)

NameTypeDescription
loraSTRING—