Pto Lr Scheduler Cosine Annealing

PtoLrSchedulerCosineAnnealing wraps PyTorch's CosineAnnealingLR and makes your learning rate ride a cosine wave down to a floor over the course of training. It's the scheduler the pack credits for pushing its CIFAR-10 ResNet from ~93% to 94.3% - the author's own example of "add cosine annealing, gain a point of accuracy." If your model plateaus and you've already checked the obvious stuff, a cosine decay to a lower learning rate is often the nudge that gets the last bit of performance out.

Why it matters

A fixed learning rate is a compromise: you want it high early to move fast, low late to settle into a minimum. Cosine annealing automates that arc. The learning rate starts at the optimizer's value, then follows eta_t = 0.5 * eta_max * (1 + cos(t_cur/t_max * pi)) down to your chosen minimum_lr as training progresses. The curve is smooth - no abrupt drops like StepLR - which is why Transformer training, including this pack's own, treats it as a default. The embedding_transformer_classification.json example pairs it with Pto AdamW; the CIFAR-10 v11 workflow added it to a ResNet.

Inputs

• optimizer (PTOPTIMIZER) - the optimizer you want to schedule. Wire Pto AdamW, Pto Adam, PtoSGD etc. here. The scheduler reads and rewrites this optimizer's learning rate each step.
• num_epochs - the total training length. This maps to PyTorch's T_max, the number of steps over which the curve descends. Set it to roughly your planned epoch count; if your actual training runs shorter or longer than this, the curve won't land where you expect.
• minimum_lr - the floor the curve descends to (PyTorch's eta_min). Default 0 is fine for most runs; a tiny non-zero floor (like 1e-6) can help if you plan to keep training past the scheduled end.

Output is a single PTLRSCHEDULER, which you feed into a training node that accepts a scheduler.

The practical gotchas

num_epochs is the knob that decides the whole shape of the curve - get it close to your real epoch count or the decay will be too fast (curve bottoms out early, wasted annealing) or too slow (never reaches the floor). If you train past the scheduled end, cosine annealing starts climbing again from the floor - that's the classic "my loss went up late in training" surprise. And remember the scheduler wraps an optimizer that's already bound to a specific model, so keep model → optimizer → scheduler all on the same run.

Installing it

Part of ComfyUI-Pt-Wrapper (HowToSD's no-code PyTorch pack, a spin-off of ComfyUI-Data-Analysis). ComfyUI Manager → search "ComfyUI-Pt-Wrapper", or:

cd ComfyUI/custom_nodes
git clone https://github.com/HowToSD/ComfyUI-Pt-Wrapper

Restart after; first boot is slow while pandas, scikit-learn, transformers, sentencepiece, peft and friends install. No model files needed for the node itself.