ComfyUI-Grok-SmartVAE

ComfyUI-Grok-SmartVAE

Version 12.0.0 - Production-Ready Video Workflow Suite

The most robust and memory-efficient VAE decoder suite for ComfyUI video workflows

---

🎯 What's Included

This package provides 5 production-ready nodes

| Node | Purpose | Best For | |------|---------|----------| | 🎬 Universal Smart VAE Decode | In-memory full-tensor decode | Short videos (<500 frames), high VRAM | | 🎬 SmartVAE Streaming Decoder | Frame-by-frame encoding to disk | Long videos (500+ frames), limited RAM | | 🎞️ SmartVAE Advanced Decoder | Streaming + temporal color correction | Professional quality, zero flicker | | 📂 Advanced Load Latent | Load saved latents with metadata | Iterative refinement workflows | | 🔍 Latent Metadata Viewer | Inspect latent files without loading | Quick library browsing |

✨ Key Features

Memory Efficiency

• ✅ Frame-by-frame decoding → 80% less RAM than standard pipeline
• ✅ Streaming video output → no full tensor in memory
• ✅ Automatic OOM recovery → tiling, batch reduction, VRAM detection
• ✅ Handles 2000+ frames on 16GB RAM systems

Production Quality

• ✅ Perfect audio sync → zero offset even on 40s+ videos
• ✅ Zero-flicker color correction → seamless chunk boundaries
• ✅ Crash recovery → resume from checkpoint
• ✅ Multiple codecs → H.264, H.265, ProRes 422, FFV1 lossless

Developer Friendly

• ✅ Automatic time scale detection → works with any VAE
• ✅ Detailed diagnostics → corrupted latent detection with actionable fixes
• ✅ Ignore warnings mode → user choice on risk tolerance
• ✅ Live thumbnails → monitor progress in ComfyUI UI

---

🚀 Quick Start

Installation

cd ComfyUI/custom_nodes
git clone https://github.com/uczensokratesa/ComfyUI-Grok-SmartVAE.git
cd ComfyUI-Grok-SmartVAE
pip install -r requirements.txt

Dependencies:

• torch>=2.0.0
• imageio>=2.31.0
• imageio-ffmpeg>=0.4.9
• numpy>=1.24.0
• safetensors>=0.4.0 (for Load Latent node)

1. In your custom_nodes folder:

   git clone https://github.com/uczensokratesa/ComfyUI-Grok-SmartVAE.git
   The node appears in category: latent/video → Universal VAE Decode (v11.1 Final)Comparison with predecessorsModel
   

Scale Detection Force Scale Dynamic Batch Reduction Auto-Tiling on OOM Loop Type Stability Rating GPT basic ✗ ✗ ✗ for ★★☆☆☆ Gemini good ✗ partial ✓ for ★★★★☆ Claude very precise ✗ ✗ ✓ for ★★★★☆ Grok v11.1 very precise ✓ full (while + adaptive) aggressive while ★★★★★

Evolution – AI collaboration storyThis journey started as a simple task: create a reliable VAE Decode node for heavy video workflows.GPT provided the first working version
Gemini added tiling and better OOM handling
Claude brought the most accurate scale detection formula
Grok introduced force_time_scale + true dynamic while-loop batch reduction
Kimi contributed extreme memory safety (disk offload, pre-allocation, aggressive cleanup)
Final polish by Claude → production-ready stability

One of the nicest examples of how different AI models can iteratively improve each other and create something better than any single one could alone.LicenseMIT – feel free to use, modify, fork. Just keep the original idea attribution (and let me know if you make something even better )Happy generating! Current version: 11.2 – First official Comfy Registry release

🎬 Universal Smart VAE Video Decode (Streaming)

The most memory-efficient way to decode large video latents directly to file.

Features

• Ultra-low RAM usage — decodes frame-by-frame, supports 2000+ frames even on 16 GB systems
• Direct disk output — no need to hold full video tensor in memory
• Audio muxing — perfect support for ComfyUI AUDIO input + manual audio_path
• Codecs: H.264, H.265, ProRes 422, FFV1 (lossless)
• Crash recovery — resumes from checkpoint if interrupted
• Thumbnail previews — live monitoring in UI
• OOM protection — automatic tiling, batch reduction, VRAM/RAM detection

When to use

• Generating long videos from AnimateDiff, LTX-Video, Mochi, Hunyuan, Cosmos, etc.
• Working with limited VRAM/RAM
• Need reliable audio sync without post-processing

Inputs

• samples (LATENT) – video latent sequence
• vae – your VAE model
• frames_per_batch – 8–32 (auto-reduces on OOM)
• audio – ComfyUI AUDIO input (from Load Audio)
• audio_path – optional direct path to .wav/.mp3
• codec, fps, output path, etc.

Outputs

• preview_thumbs (IMAGE) – last few thumbnails for monitoring
• video_path (STRING) – final file path (with audio if provided)

Enjoy massive video workflows without OOM crashes! 🚀

🛡️ Ignore Warnings Mode (v1.0.9)

Problem: When the sampler runs out of VRAM during long video generation, it produces corrupted latents (100% NaN values) but doesn't throw an error (silent OOM). Users then face black frames or workflow crashes without understanding why.

Solution: The Streaming node (and VAE) now includes a 3-tier ignore_warnings safety system:

How It Works

1. Validation on Decode: Before decoding, the node checks if the latent contains NaN values (corruption indicator)
2. Detailed Diagnostics: Shows exactly which frames are corrupted and corruption percentage
3. User Choice: Instead of hard-failing, users can choose their risk tolerance:

Basic Usage

Generate long video:

[Empty Latent] (1200 frames, 768x512)
  ↓
[KSampler] (LTX-Video 2 model)
  ↓
[🎬 SmartVAE Streaming Decoder]
  frames_per_batch: 8
  codec: h264
  fps: 24
  ↓
Output: video_1234567890.mp4

With color grading:

[Empty Latent] → [KSampler] → [Save Latent]
  ↓
[📂 Advanced Load Latent] → [Load Image] (reference)
  ↓                           ↓
[🎞️ SmartVAE Advanced Decoder]
  anti_color_bleed: ON
  correction_strength: 0.18
  ↓
Output: video_graded.mp4 (zero flicker, consistent color)

---

📚 Node Documentation

🎬 SmartVAE Streaming Decoder

Purpose: Memory-efficient video generation with direct disk output.

Key Parameters:

| Parameter | Default | Description | |-----------|---------|-------------| | frames_per_batch | 8 | Frames per decode batch (auto-reduces on OOM) | | overlap_frames | 2 | Temporal overlap for seamless stitching | | fps | 24 | Output frame rate | | codec | h264 | h264 (compatible) / h265 (smaller) / prores / ffv1 (lossless) | | audio | None | ComfyUI AUDIO input (perfect sync) | | ignore_warnings | none | Corrupted latent handling (none/minor/all) |

Advanced Parameters:

• force_time_scale (0=auto, 8 for LTX-Video)
• enable_tiling (auto-enables on OOM)
• resume_on_crash (checkpoint recovery)

Recommended Settings by Video Length:

| Length | Frames | frames_per_batch | Expected RAM | |--------|--------|------------------|--------------| | Short (5-15s) | 120-360 | 16-32 | 4-8 GB | | Medium (20-40s) | 480-960 | 8-16 | 8-12 GB | | Long (50s+) | 1200+ | 4-8 | 12-16 GB | | Ultra (2min+) | 3000+ | 4 | 16+ GB |

Common Issues:

Q: Video is all black frames

• Cause: Corrupted latent from sampler (OOM during generation)
• Solution: Check console for NaN detection warning. Reduce video length by 40%, lower CFG scale, or enable CPU offload in sampler.

Q: Audio desync (was 1s offset)

• Solution: Fixed in v1.0.10! Upgrade to latest version.

Q: Out of memory during decode

• Solution: Node auto-reduces batch size. If still failing, manually set frames_per_batch: 4 and enable_tiling: ON.

---

🎞️ SmartVAE Advanced Decoder

Purpose: Streaming decoder + temporal color correction for professional output.

New Feature: Ramped Temporal Color Correction v2.3.1

• Zero flicker at chunk boundaries
• Gradual cross-fade correction (imperceptible transitions)
• Tested: 50s+ videos, LTX-Video 2 compatible
• Overhead: ~2-5% processing time

Key Parameters:

| Parameter | Default | Description | |-----------|---------|-------------| | anti_color_bleed | True | Seamless temporal color matching | | calibration_image | None | Reference image for color grading | | correction_strength | 0.18 | 0.10-0.15 (subtle), 0.18 (balanced), 0.20-0.25 (strong) | | cross_fade_frames | 12 | Fade duration (0.5s @ 24fps) | | ema_momentum | 0.93 | Color evolution smoothing (0.7-0.98) |

Three Correction Modes:

1. Temporal only (anti_color_bleed: ON, no reference)

   • Auto-matches chunk boundaries
   • Natural color evolution

2. Reference only (anti_color_bleed: OFF, reference image provided)

   • Matches first chunk to reference
   • Rest evolves naturally

3. Temporal Match (BOTH enabled - recommended)

   • Reference sets initial anchor
   • Temporal ensures continuity throughout

Parameter Guide by Use Case:

| Use Case | correction_strength | cross_fade_frames | ema_momentum | |----------|---------------------|-------------------|--------------| | Subtle/natural | 0.10-0.15 | 8 | 0.93 | | Balanced (default) | 0.18 | 12 | 0.93 | | Cinematic/strong | 0.20-0.25 | 12-18 | 0.93-0.95 | | Dynamic lighting | 0.15-0.18 | 8-12 | 0.70-0.85 | | Locked to reference | 0.20-0.25 | 18-24 | 0.95-0.98 |

Troubleshooting:

• Still seeing color jumps? → Increase correction_strength to 0.25-0.30, add reference image
• Colors look artificial? → Decrease correction_strength to 0.10-0.15
• Not tracking dynamic lighting? → Decrease ema_momentum to 0.7-0.85

---

📂 Advanced Load Latent

Purpose: Load saved latents for iterative workflows without re-sampling.

Features:

• ✅ Auto-scan output/latents/ directory
• ✅ Dropdown selection with refresh button
• ✅ Metadata display (shape, seed, timestamp, format)
• ✅ PyTorch 2.6+ compatible (safetensors primary, pickle fallback)
• ✅ Manual path override

Workflow Example: Iterative Color Grading

Step 1: Generate & Save
  [Empty Latent] (1200 frames) → [KSampler] (seed=42)
    ↓
  [Save Latent] → output/latents/base_1200f_s42.latent

Step 2: First Attempt
  [📂 Advanced Load Latent] → base_1200f_s42.latent
    ↓
  [🎞️ SmartVAE Advanced Decoder] (strength=0.15)
    ↓
  Output: video_001.mp4 (too subtle)

Step 3: Iterate (no re-sampling!)
  [📂 Advanced Load Latent] → base_1200f_s42.latent (same file)
    ↓
  [🎞️ SmartVAE Advanced Decoder] (strength=0.25, +reference)
    ↓
  Output: video_002.mp4 (perfect!)

Time saved: ~5 minutes per iteration × 3-4 tests = 15-20 minutes ✅

Supported Formats:

• .latent (ComfyUI default)
• .safetensors (modern, secure)

Outputs:

• samples (LATENT) → connect to decoder
• metadata_json (STRING) → shape, seed, timestamp, format

---

🔍 Latent Metadata Viewer

Purpose: Quick inspection without loading full tensor (10GB+ files).

Use Case: Browse large latent collections, organize by seed/date/format.

Outputs:

• shape → "[1, 4, 121, 80, 120]"
• seed → "42"
• timestamp → "2025-02-26T15:30:00"
• format → "safetensors" or "pickle"

---

🛡️ Safety Features

Corrupted Latent Detection (v1.0.9+)

Problem: When samplers run out of VRAM, they produce corrupted latents (NaN values) without throwing errors. Users get black frames without understanding why.

Solution: Pre-decode validation with user choice:

🚨 CORRUPTED LATENT DETECTED!
   NaN values: 4,567,890 / 49,152,000 (9.29%)
   Affected frames: 856 to 1200
   
💊 RECOMMENDED FIXES:
   1. REDUCE VIDEO LENGTH: 1201 → 720 frames (most effective)
   2. REDUCE RESOLUTION
   3. Lower CFG scale (7.0 → 3.5)
   4. Enable CPU offload in sampler

Ignore Warnings Modes:

| Mode | Behavior | Use Case | |------|----------|----------| | none (default) | Stop with error + solutions | Safe, prevents wasted time | | minor | Continue if <10% corrupted | Partial recovery (some black frames OK) | | all | Force decode anyway | Desperate (high crash risk) |

User makes informed decision instead of blaming the node.

---

🐛 Known Limitations

LTX-Video 2 Architecture Limitation

Issue: LTX-Video 2 has a hard architectural limit that can cause latent corruption on long videos.

NOT a node bug - this is a model limitation:

• Symptom: NaN values in latent after ~1000-1200 frames
• Root cause: LTX-2 model architecture (not VRAM issue)
• Solution: Keep videos under 40-45s (~1000 frames @ 24fps)

Wrong diagnosis (old README):

❌ "Silent VRAM death" → implied decoder bug

Correct diagnosis (updated):

✅ "LTX-2 architectural limit" → model constraint

If you hit this limit:

1. Reduce video length to ~1000 frames
2. Split into multiple segments
3. Use different model (CogVideoX, AnimateDiff, etc.)

---

📊 Performance Benchmarks

Test System: RTX 3090 (24GB VRAM), 64GB RAM

| Scenario | Standard Pipeline | SmartVAE Streaming | Savings | |----------|-------------------|-------------------|---------| | 30s video (720 frames, 768x512) | 18.2 GB RAM | 3.8 GB RAM | 79% ↓ | | 60s video (1440 frames, 768x512) | 36.4 GB RAM (OOM) | 7.2 GB RAM | 80% ↓ | | With color correction | N/A | +2-5% time | Minimal | | Audio muxing (old) | 45s (re-encode) | 4s (stream copy) | 91% ↓ |

---

🎨 Example Workflows

Basic: Long Video Generation

[Empty Latent] (1200 frames, 768x512, batch=1)
  ↓
[KSampler] (LTX-Video 2 VAE, steps=30, cfg=3.5)
  ↓
[🎬 SmartVAE Streaming Decoder]
  frames_per_batch: 8
  fps: 24
  codec: h264
  ↓
Output: video.mp4 (50s, perfect quality)

Advanced: Color-Graded with Reference

[Load Image] → reference.png
  ↓
[Empty Latent] → [KSampler] → [Save Latent] → base.latent
  ↓
[📂 Advanced Load Latent] → base.latent
  ↓        ↓
  ↓      reference.png
  ↓        ↓
[🎞️ SmartVAE Advanced Decoder]
  anti_color_bleed: ON
  calibration_image: (connected)
  correction_strength: 0.18
  cross_fade_frames: 12
  ↓
Output: graded.mp4 (zero flicker, matches reference color)

Pro: Iterative Refinement

[KSampler] → [Save Latent] → base_s42.latent

[📂 Advanced Load Latent] → base_s42.latent
  ↓
[🎞️ Advanced Decoder] → try strength=0.15 → video_v1.mp4 (too subtle)

[📂 Advanced Load Latent] → base_s42.latent (same file)
  ↓
[🎞️ Advanced Decoder] → try strength=0.25 → video_v2.mp4 (perfect!)

No re-sampling! Instant iteration!

---

🏗️ Technical Details

Architecture

Base Class: SmartVAE_StreamingDecoder

• Frame-by-frame VAE decode
• Dynamic batch reduction (while loop)
• Automatic tiling on OOM
• StreamingVideoWriter (imageio backend)

Advanced Class: SmartVAEAdvancedDecoder (inherits base)

• Adds ramped temporal color correction
• EMA-based color evolution tracking
• Chunk boundary cross-fade

Load Latent: Standalone utility

• Safetensors primary (PyTorch 2.6+ compatible)
• Pickle fallback (legacy support)
• Metadata extraction without full load

Audio Sync Fix (v1.0.10)

Problem: 1s audio offset on long videos.

Root cause: Video re-encoding during mux → encoder delay → PTS misalignment.

Solution:

# Before (broken):
ffmpeg -i video.mp4 -i audio.wav -c:v libx264 ...  # Re-encodes!

# After (fixed):
ffmpeg -fflags +genpts -i video.mp4 -i audio.wav \
  -c:v copy \                    # Stream copy (no re-encode)
  -vsync cfr \                   # Constant frame rate
  -af asetpts=PTS-STARTPTS \     # Reset audio PTS
  -itsoffset 0 ...               # Force video t=0

Result: Zero offset, 10x faster muxing, lossless quality.

---

🤝 Credits & Evolution

This project is a collaboration across multiple AI models:

• GPT → Foundation sliding-window + overlap
• Gemini → Safety-first tiling fallback
• Claude → Precise temporal scale detection + QA
• Grok 4.2 → Dynamic batch reduction + ramped color correction
• Kimi → Memory-safety patterns
• OpenAI Codex → Bug fixes + resume logic

Key Contributors:

• uczensokratesa - Project lead
• Community feedback - Feature requests & testing

One of the best examples of iterative AI collaboration creating something better than any single model could alone.

---

📄 License

MIT License - Feel free to use, modify, fork.

Attribution appreciated but not required.

---

🔗 Links

• Repository: https://github.com/uczensokratesa/ComfyUI-Grok-SmartVAE
• Issues: https://github.com/uczensokratesa/ComfyUI-Grok-SmartVAE/issues
• ComfyUI Registry: Coming soon

---

📝 Changelog

v11.5.0 (Branch: Twonewnodes)

• ✨ NEW: Advanced Load Latent node (PyTorch 2.6+ compatible)
• ✨ NEW: Latent Metadata Viewer
• ✨ NEW: SmartVAE Advanced Decoder v2.3.1 (ramped color correction)
• 🐛 FIX: Audio sync issue (1s offset eliminated)
• 🐛 FIX: Complete __init__ in Advanced Decoder (edge case crashes)
• ⚡ OPT: Optimized stats extraction (~5-10% speedup)
• 📝 DOC: Complete README rewrite with accurate diagnostics

v1.0.10 (Streaming Decoder)

• 🐛 FIX: Critical audio sync bug (stream copy + PTS normalization)
• 🐛 FIX: NaN detection with actionable error messages
• ✨ NEW: Ignore warnings mode (none/minor/all)
• 📝 DOC: Clarified LTX-2 architectural limitation (not VRAM bug)

v1.0.9

• ✨ NEW: Corrupted latent detection with diagnostics
• ✨ NEW: Ignore warnings safety system
• 🐛 FIX: NaN handling in normalization

v11.2

• ✨ First ComfyUI Registry release

• ✨ Production-ready stability

  v12.5

• ✨ Rewrite code for Advanced Decoder by Claude AI.

• ✨ Independent from Streaming Decoder to really have ability to color correction.

---

v13.0

• ✨Rewrite code for finding dimensions of video Decoder by Gemini and Claude AI. Full Dimension Support and Safe Error Handling (Universal Smart VAE Decode)This update brings critical improvements to the VAE decoding pipeline, making the Universal Smart VAE Decode node the most stable and reliable video decoding solution for ComfyUI. We have eliminated long-standing issues with non-standard aspect ratios and introduced a powerful new error-recovery system. What's New?1. Complete Freedom in Resolution and Aspect Ratio (No more ghosting, black bars, or corrupted output)Problem: Previous versions (and many other decoders) attempted to infer the output shape (Frames, Height, Width) by sorting dimensions numerically. This approach broke when generating long videos in lower resolutions — for example, when the frame count exceeded the pixel dimensions of height or width.Solution: We completely rewrote the _normalize_output function. The new algorithm no longer guesses dimensions based on size. Instead, it reliably detects color channels (RGB/RGBA) by position and dynamically assembles tensors into the correct [Frames, Height, Width, Channels] format.Result: You can now generate videos in any resolution, aspect ratio, and duration without decoding errors.2. New ignore_warnings Mode (Crash Protection for VRAM-intensive Generations)Video generation is extremely memory-intensive. Occasionally, a brief VRAM spike at the very end of a long render can corrupt a small portion of the latent (introducing NaN values). Most nodes would crash and discard hours of work. We changed that.The new ignore_warnings parameter offers three safety levels: none (Default / Safe) Stops execution on error but provides a detailed console report, including the exact percentage of corrupted data, affected frame ranges, and practical advice on how to prevent the issue. minor (Recommended for video recovery) Forces the VAE to continue if damage is below 10%. Corrupted frames (usually at the end) will be rendered as black frames, but the node completes successfully and saves the intact portion of the video. all (Maximum Force) Ignores all errors and pushes data through the VAE at all costs. Higher risk of completely black output, but gives the best possible chance of salvaging the generation.

Under the Hood (Technical Changes)Replaced dimension-based sorting and tensor.permute logic with robust shape verification using tensor.shape[-1] in (3, 4). Damaged latents (NaN / Inf) are now safely sanitized before decoding using torch.nan_to_num(tensor, nan=0.0, posinf=1.0, neginf=0.0). This prevents GPU driver crashes. Enhanced memory cleanup logic (gc.collect() + torch.cuda.empty_cache()) in critical failure scenarios.

Happy generating! 🎬✨