Optimizer Config Adafactor

Adafactor is the optimizer you pick when you're fighting for every gigabyte of VRAM. Where Adam-family optimizers keep a couple of full-size statistics per weight, Adafactor factorizes them to use dramatically less memory - the trade being that it's a bit slower and a bit fussier to tune. OptimizerConfigAdafactor is the node that sets it up for a Flux Trainer run, with its distinctive knobs surfaced.

It's the classic choice for full fine-tuning and for large models on modest cards, which is exactly the situation where the memory savings pay off. For a normal LoRA on a card with headroom, you probably don't need it - AdamW8bit is simpler. Adafactor earns its place when memory is the binding constraint.

How it works

The node outputs optimizer_settings (type ARGS) for the Init node. Adafactor's behavior hinges on a few booleans that interact, and getting them consistent is most of the battle:

• relative_step (default false) - when true, Adafactor computes its own step size internally rather than using your learning rate. This is Adafactor's own "adaptive" mode.
• scale_parameter (default false) - scales the learning rate by parameter magnitude.
• warmup_init (default false) - warms up the internal step size; typically paired with relative_step.
• clip_threshold (default 1) - clips update magnitude for stability.
• lr_scheduler (default constant_with_warmup) - note the default already includes warmup, which suits Adafactor.
• min_snr_gamma (default 5) - standard loss weighting.

The key decision is whether you drive it with a fixed learning rate (set on the Init node) or let it run adaptive with relative_step on. Mixing those up is the usual source of grief.

When to reach for it

Pick Adafactor when VRAM is the wall - full Flux fine-tuning, or a large train on a smaller card - and you're willing to trade some speed for it. If you've got memory to spare, the simpler AdamW8bit path is less fiddly and usually just as good for LoRA work.

Installing the pack

ComfyUI Manager: search ComfyUI Flux Trainer, install, restart. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/kijai/ComfyUI-FluxTrainer
pip install -r ComfyUI-FluxTrainer/requirements.txt

Torch 2.4.0+ recommended, plus kijai's ComfyUI-KJNodes for the example workflows.

Common issues

relative_step and a fixed LR fight each other. If relative_step is on, Adafactor sets its own step size and your learning rate is largely ignored - which surprises people who then can't understand why LR changes do nothing. Decide on one mode: fixed LR with relative_step off, or adaptive with it on (usually alongside warmup_init).

It's slower - that's expected. Adafactor trades speed for memory. If a run feels sluggish compared to Adam, that's the deal, not a bug. Only accept the slowdown if you actually need the VRAM savings.

Convergence looks different. Adafactor doesn't behave identically to Adam, so a settings table written for AdamW won't transfer cleanly. Lean on the pack's validation nodes to check progress rather than assuming Adam-era numbers apply.