Illustrious SDXL LoRA Train

LoRA training and image generation have drifted into separate tools. Ask how to train a LoRA and you'll get kohya, OneTrainer or ai-toolkit; ask what to generate with and you'll get ComfyUI. The ComfyUI trainer nodes that existed mostly wrapped a training script rather than implementing one - nicer front ends over the same shell command.

IllustriousSDXLLoRATrain bets the other way: it trains a UNet LoRA in-process, with Diffusers and PEFT, inside the same Python that's drawing your previews. No subprocess, no separate environment, no kohya install. An unusual architecture - which is both why it's interesting and why you should be careful with it.

What happens when you queue it

First, enabled has to be true - the default is false and the node returns NOT_RUN with "Training disabled. Review captions, then enable explicitly" otherwise. The pack is built so that loading a workflow can never start a training run by accident.

Then: the caption audit has to pass (every image, non-empty caption, or it raises); the model directory is validated; a process-wide lock is taken; and the pipeline loads local-only (local_files_only=True), so it will never fetch a base model for you. VAE and both text encoders stay frozen, an adapter goes on the UNet only (to_q, to_k, to_v, to_out.0, Gaussian init), and training starts.

The loop is the standard SDXL objective: VAE-encode the batch, sample a random timestep, add noise with a DDPMScheduler, predict, backprop. On top sit the options-node mechanics: buckets so a batch is homogeneous in resolution, accumulation with the last partial window weighted correctly instead of quietly discounted, clipping at 1.0, Min-SNR weighting, noise offset, cosine or constant schedule with warmup. One detail worth appreciating: ComfyUI executes nodes under torch.inference_mode, so the trainer explicitly turns inference mode off and grad on. That's a big part of why "train inside ComfyUI" is harder than it sounds.

Saving runs through Diffusers' Kohya conversion, so the output is a .safetensors ComfyUI loads like any other LoRA.

The inputs that matter

enabled - true, deliberately. checkpoint is an enum that starts with DIFFUSERS_DIRECTORY and otherwise lists the .safetensors files in ComfyUI/models/checkpoints. sdxl_config_or_model_directory (default diffusers/illustrious_sdxl) is the path relative to ComfyUI/models holding either a complete Diffusers SDXL pipeline (model_index.json, unet/, vae/, both text encoders and tokenizers, scheduler) or the configs and tokenizers that pair with your single-file checkpoint. Either way the base weights must be local and safetensors-only; .py, .pkl, .pt, .bin and .ckpt files inside that directory are rejected outright.

output_name is a bare name (letters, digits, _, -), not a path. epochs, resolution (256–1024, multiples of 64), batch_size and gradient_accumulation set the arithmetic of the run. Then rank, learning_rate, precision (bf16 or fp32), save_every_epochs, gradient_checkpointing, and seed - which carries control_after_generate, so set it to fixed before you queue if you want to reproduce the run, not after.

The optional advanced_options input is where the real knobs live; wire IllustriousTrainingOptions into it. Leave it unwired and you don't get the documented defaults - you get the trainer's bare fallbacks: no buckets, no epoch samples, 75-token captions, constant learning rate, repeat 1. That's the single most confusing thing about this pack, and it's silent.

Outputs are lora_path, the string path to final.safetensors, and training_report, JSON with the run state. The report is what you feed to IllustriousEpochSamples. Everything lands under ComfyUI/output/illustrious_native/<dataset_run>/<output_name>_<hex>/, alongside epoch_###.safetensors files at your save interval and a training.json tracking the run.

Getting close to the right settings

The SDXL tables have said 16/16 or 32/16 for rank/alpha for two years, LR between 1e-5 and 1e-4, and roughly 100–200 steps per image. The pack's defaults - rank 16, alpha 16, LR 1e-4, bf16, gradient checkpointing - sit inside all of that. Two habits worth keeping: save intermediate epochs, because the last one is usually not the best, and prove the pipeline on a handful of images at one epoch before launching a 20-epoch job.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/katorikonoe-ai/ComfyUI-IllustriousLoRATools
pip install -r ComfyUI-IllustriousLoRATools/requirements.txt

Or Manager, searching ComfyUI-IllustriousLoRATools (Illustrious LoRA Tools). This node needs the whole dependency list - diffusers, transformers, peft, accelerate, bitsandbytes and the rest - into ComfyUI's own Python, preserving the host CUDA PyTorch. pip install into a working ComfyUI environment is the riskiest step in this entire workflow; back up an older install before replacing it.

Where people get burned

CUDA, or nothing. There is no CPU training path; without a CUDA GPU it raises. bf16 additionally requires your GPU to support it, otherwise pick fp32 and expect to feel it. Batch 1 with accumulation 4 is not a low-VRAM guarantee. Close your generation graphs first - other loaded models eat the same VRAM.

Interrupted is not finished. If the run dies mid-way, the pack writes INCOMPLETE.safetensors with an explicit flag. It is not a completed adapter and there is no optimizer resume, so don't ship it and don't expect to continue from it.

The base model has to match the config. Epsilon versus v-prediction comes from the scheduler config in your model directory, not from the checkpoint filename. A v-prediction base trained against an epsilon config trains the wrong objective, and it will look fine right up until you use the LoRA.

Unvalidated means unvalidated. The author says plainly that full pretrained SDXL CUDA training and the tagger quality still need external validation, that the release workstation was a 2 GB MX250, and that TESTING.md documents measured checks "not marketing claims." Take that at face value: this is an interesting, honest, early implementation, not a drop-in kohya replacement. If you're training your first LoRA and you want the shortest path to a good one, ai-toolkit or the kohya GUI is still the boring correct answer. Reach for this one when you want the whole loop - tag, review, train, sample - inside one graph, and you're willing to test.

Once you have final.safetensors, copy it to ComfyUI/models/loras/, load it with a compatible Illustrious-family base, use CLIP skip 2 (Clip Set Last Layer at -2), and prompt your trigger word. Then check the epoch samples - the middle epochs are usually better than the finale.