ComfyUI-Gradual-IC-LoRA

ComfyUI-Gradual-IC-LoRA [beta]

Beta — functionally validated, design caveat documented. All four smoke tests pass against live LTX-2.3 22B. Gradual within-clip LoRA ramps work, but read "Gradual strength: what to expect" before you build a workflow around a body-size or other spatially-drastic IC-LoRA — the model's temporal-consistency prior fights some effects more than others.

Gradually modulate the strength of a user-trained IC-LoRA (video conditioning, reference tokens at negative RoPE positions) over the output clip duration in a single continuous LTX-2.3 sampling run.

The LoRA is applied as a runtime, per-token-masked adapter — weights are never merged. Reference tokens are held at a constant strength (so the conditioning stays fully readable for the whole run); output tokens follow the curve, sampled per latent frame.

Install

Folder-drop into ComfyUI/custom_nodes/ and restart ComfyUI. No extra Python deps beyond what ComfyUI already ships (torch, numpy, Pillow).

ComfyUI/custom_nodes/ComfyUI-Gradual-IC-LoRA/

Nodes

IC Strength Curve

Custom canvas widget. X = normalised clip time 0..1, Y = strength multiplier (default range -0.5 .. +2.0, adjustable via the y_min/y_max widgets).

• Click empty space to add a keyframe; drag to move; right-click or double-click a keyframe to delete (minimum 2 kept).
• Interpolation: click the curve: chip to cycle monotone → catmull → linear.
• Presets: constant, linear, ease in, plateau step.
• Output: IC_CURVE (serialized keyframes + mode). Resampling to frames happens in IC-LoRA Curve Apply, so the curve node never needs the frame count.
• State serializes into the workflow JSON via a hidden backing widget. For headless / API use, set preset_fallback instead of drawing.

IC-LoRA Curve Apply

Patches an LTX-2.3 MODEL with the IC-LoRA as a masked runtime adapter.

Inputs

| input | notes | |---|---| | model | LTX-2.3 MODEL, after all static LoRAs | | video_latent | the final video-only latent; used only as a timing reference; passed through | | lora_name | IC-LoRA safetensors (video-branch keys) | | curve | IC_CURVE | | strength_global | multiplies the curve (output tokens only) | | reference_strength | constant mask value for reference tokens (default 1.0) | | ref_token_count | reference token count; set this unless ic_layout is wired | | ic_layout (opt) | layout metadata from your IC-LoRA inference node, if a handle exists; overrides ref_token_count | | num_latent_frames (opt) | 0 = auto from video_latent temporal dim | | rendered_frames (opt) | 0 = ignore; else cross-checks via the LTX 8× rule | | ref_tokens_first (opt) | are reference tokens the leading block of the sequence? default true | | token_order (opt) | frame_major (default) or spatial_major token flatten order | | low_vram (opt) | compute B(A·x) in sequence chunks |

Outputs: model (patched), video_latent (passthrough, for stacking), schedule_info (STRING: resolved frame count, per-frame strengths, ref token count, patched-module count).

Mechanism

out = base(x) + s ⊙ (α/r)·B(A·x)

s is a per-token scalar mask broadcast over the hidden dim:

• Reference tokens → reference_strength (constant).
• Output tokens → curve sampled at the token's latent-frame index, times strength_global.

Latent frames follow LTX 8× temporal compression / 8n+1: latent_frames = (rendered_frames − 1) / 8 + 1.

The mask rule is uniform across all patched projections (Q, K, V, out, FFN) — these are per-token Linears, so one rule covers everything. No Q-only casing.

Token-layout hard fail. At each forward we check seq_len == ref_token_count + output_tokens and that output_tokens divides evenly by num_latent_frames. On mismatch the node raises with a detailed message rather than silently mis-aligning the curve to the wrong frames (the worst-case failure mode). The mask is built once per unique sequence length and cached.

Video modules only. Patch targets are derived from the LoRA's own keys via ComfyUI's comfy.lora key mapping, so only the modules the (video-branch) IC-LoRA touches get patched. Any audio-branch keys are skipped with a warning.

IC Curve Preview (optional)

Renders the resolved per-frame strength as an IMAGE (pure-PIL) with frame indices and second markers at a given fps. Useful QA before a long render.

Wiring

[IC Strength Curve] ──curve──┐
                             ▼
[LTX MODEL (post static LoRAs)] ──model──► [IC-LoRA Curve Apply] ──model──► [Sampler]
[final video-only LATENT] ──video_latent──►                    └─video_latent─► (passthrough)

• Place IC-LoRA Curve Apply last in the MODEL chain, after all static LoRAs, immediately before the sampler.
• Feed it the final video-only latent.
• Set ref_token_count (or wire ic_layout) to your IC reference-token count.

Stacking two Curve Apply nodes (different LoRAs) composes additively — each wraps the previous adapter — and video_latent passes straight through.

Smoke test results (all PASS, RTX 5090, LTX-2.3 22B fp8)

1. Gate — const 1.0 matches normal IC-LoRA ✓ (9.6% latent rms, fp8 floor; frames visually identical). Sampler switched to euler for a noise-free gate (euler_ancestral injects unseeded per-step noise; two identical runs differ ~70% in latent space).
2. Reference bleed (const 0.0 / ref 1.0) ✓ — output vs base = 8.3% (fp8 floor). No meaningful reference-token bleed → phase-2 attention bias not needed.
3. Linear ramp 0→1 (the feature) ✓ — per output latent frame, early frames track base and late frames track full with a clean monotonic crossover → per-token time-curve masking + frame ordering correct.
4. Stacking ✓ — two chained Curve Apply nodes at 0.5 each vs a single at 1.0 = 9.8% (fp8 floor) → composes additively.

Three application bugs and one cross-run leak were found and fixed during smoke testing (see commit history). Driver at smoketest/run_gate.py.

Verified layout for the test workflow

seq_len 15840 = 18 latent frames × 880 tokens; output = 9 latent frames (num_latent_frames=9), reference = 7920 tokens (ref_token_count=7920), token_order=frame_major, ref_tokens_first=True. Spatial token count is constant at 880 for the 1280‑long‑side resize this workflow uses (invariant across aspect ratios).

Gradual strength: what to expect

A within-clip IC-LoRA ramp is not the same as rendering two separate clips at different strengths and cross‑fading. LTX enforces a strong temporal‑consistency prior: the model commits to roughly one coherent body, style, or scene state per continuous shot. How much this matters depends on your LoRA:

| LoRA type | Example | Temporal‑consistency impact | |---|---|---| | Drastic spatial change | body‑size, object insertion/removal | Strong. The start of a ramp reads diluted vs a constant render at the same strength, because the weaker tail pulls the whole clip toward the average. You may need to push strength well above your usual constant‑render value or sacrifice gradualness for impact. | | Style / colour / lighting | film‑grade, painterly, desaturation | Mild to none. These have no hard consistency prior — the model readily blends strength levels, and a smooth ramp tracks faithfully. | | Fine‑detail / texture | skin detail, fabric, sharpening | Mild. Generally tracks well; check for temporal flicker at very low strengths. |

Mitigation for spatially‑drastic LoRAs (body‑size, etc.): an induced camera motion trick — apply a subtle synthetic pan/zoom to the input, render with the ramp, then reverse the pan on the decoded output — can approximately double the visible gradient span by relaxing the static‑body prior. A production‑ready helper ships in smoketest/fake_motion.py. See smoketest/FINDINGS.md for the full quantitative matrix.

If you need both a fully‑strong start AND a fully‑weak end for a spatially‑drastic LoRA, the reliable option is two separate constant‑strength renders cross‑faded in edit — that sidesteps the model's consistency prior entirely.

Limitations

• torch.compile unsupported (dynamic per-forward mask + object-patch wrappers). SageAttention keeps working — we only wrap projection Linears, not the attention compute.
• Video-only. Audio-branch keys are skipped.
• fp8 accuracy floor. On fp8‑quantised LTX models the baseline merges the LoRA into fp8 weights and requantizes; our adapter applies it separately at higher precision. A constant‑1.0 curve differs from the standard loader by ~9–10% latent rms — visually imperceptible, but it means the gate can't be bit‑exact on fp8 hardware. On fp16/bf16 models we expect exact matching.
• Temporal‑consistency dilution for spatially‑drastic LoRAs (see "Gradual strength: what to expect").
• Composes with other ModelPatcher-based patches via model.clone() + add_object_patch. Upstream weight‑patches (e.g. a distilled LoRA) are re‑keyed onto the wrapped .base modules automatically.

Assumptions (verified against live LTX-2.3)

These were confirmed against live LTX-2.3 during smoke testing:

• Token flatten order is frame_major (all spatial tokens of frame 0, then frame 1, …). ✓
• Reference tokens are the leading block (ref_tokens_first = true). ✓
• comfy.lora.model_lora_keys_unet recognises PEFT‑style .lora_A/.lora_B keys for LTX-2.3. ✓
• comfy.lora.load_lora returns comfy.weight_adapter.LoRAAdapter objects (not legacy ("lora", payload) tuples) on this ComfyUI version. ✓

Tests

python3 tests/test_curve.py — pure-Python checks of the curve math, including the smoke-test-1 invariant at the curve level (constant 1.0 → all-ones for any frame count). No torch/numpy required.