ComfyUI Extension: ComfyUI-Tensor-Prism-Node-Pack
Advanced model merging and enhancement nodes for ComfyUI
Custom Nodes (0)
README
ComfyUI-Tensor-Prism-Node-Pack
Developer Notes
IMPORTANT: You will have to do a regular ModelMergeSDXL for the block layer merging, since it's currently still being fixed.
My First ComfyUI Node Pack, sort of vibe-coded with Gemini 2.5 Flash and Claude 4. Feel free to publish the models you make and link them to me I'd like to be able to see the models, and see what they're about to see if I need to add more nodes or if the nodes are good and make really good quality checkpoint models. This is also a node pack for those familiar with merging models.
TensorPrism ComfyUI Node Pack
Advanced model merging and enhancement nodes for ComfyUI, providing sophisticated techniques for blending, enhancing, and manipulating Stable Diffusion models with GPU-optimized memory management.
Features
Core Merging Nodes
- Main Merge: Advanced model merging with multiple interpolation methods (linear, slerp, cosine, directional, frequency, stochastic)
- Prism: Fast spectral merging with frequency-based blending techniques including spectral blend, frequency bands, magnitude weighting, adaptive mixing, and harmonic merging
- SDXL Block Merge: Granular control over individual SDXL UNet blocks with support for TIES merging
- SDXL Advanced Block Merge: GPU-optimized block merging with intelligent memory management for any GPU size (including 12GB and smaller cards)
- Epsilon/V-Pred Block Merge: Granular block-level model merging with V-Pred/Epsilon conversion and individual control for input blocks 0-8, middle blocks 0-2, and output blocks 0-8
Conversion and Processing Nodes
- Epsilon/V-Pred Converter: Pure converter between V-Prediction and Epsilon prediction types with configurable conversion strength and smart layer targeting
- V-Pred/Epsilon Converter: Advanced prediction type conversion with critical layer identification (output blocks, middle blocks) and secondary layer processing (time embedding, input blocks)
Advanced Mask System
- Model Mask Generator: Create sophisticated masks for selective model merging with layer-based, block-based, attention-only, feedforward-only, custom patterns, random sparse, and depth gradient options
- Weighted Mask Merge: Apply masks to control where and how models are blended with tensor-level precision
- Model Key Filter: Memory-efficient filtering of model parameters with batch processing for large models
- Mask Blender: Combine multiple masks with various blending modes (Add, Multiply, Max, Min, Linear Blend, Exponential Blend)
CLIP Processing
- Advanced CLIP Merge: Sophisticated CLIP merging with multiple interpolation methods:
- Linear: Standard interpolation
- SLERP: Spherical Linear Interpolation for smooth blending
- Cosine: Cosine-based smooth transitions
- Weighted Average: Magnitude-based weighting
- Spectral Blend: Frequency domain blending with separate magnitude/phase control
- Layer-specific bias controls for attention, feedforward, embedding, and normalization layers
- Norm preservation options for maintaining model stability
Model Transformation
- Model Weight Modifier: Memory-efficient weight modification with operations like multiply, add, set value, clamp magnitude, and scale max absolute value
Utility Nodes
- Checkpoint Reroute + Notes: Convenient rerouting node for MODEL, CLIP, and VAE connections with optional note-taking functionality for workflow organization and documentation
Advanced Features
- GPU-Optimized Memory Management: Intelligent memory allocation and cleanup for cards with limited VRAM
- Batch Processing: Process large models in memory-efficient batches
- Multiple Merging Algorithms: Including SLERP, frequency domain blending, stochastic merging, and TIES merging
- Spectral Analysis: Frequency-based model analysis and merging
- Adaptive Precision: Automatic FP16/FP32 selection based on available memory
- Cross-Device Compatibility: Works with CUDA, MPS, and CPU backends
- Prediction Type Conversion: Automatic conversion between Epsilon and V-Prediction model types with mathematical accuracy
- Advanced Memory Context Management: Context managers for automatic cleanup and memory optimization
- Workflow Organization: Built-in documentation and organization tools for complex merging workflows
Installation
Method 1: Git Clone
cd ComfyUI/custom_nodes/
git clone https://github.com/NoxTheCubeman/ComfyUI-Tensor-Prism-Node-Pack.git
Method 2: ComfyUI Manager
Note: Not available yet there.
- Open ComfyUI Manager
- Search for "TensorPrism"
- Click Install
Method 3: Manual Installation (OPTIONAL)
- Clone or download this repository
- Place the entire folder in your
ComfyUI/custom_nodes/
directory - Restart ComfyUI
Usage
Basic Model Merging
Use the Main Merge node to blend two models with various interpolation methods:
- Connect two MODEL inputs
- Adjust merge ratio (0.0 = Model A only, 1.0 = Model B only)
- Choose merging method based on your needs:
- Linear: Standard interpolation
- SLERP: Spherical interpolation for smoother blending
- Cosine: Smoother transitions with cosine curve
- Directional: Vector-based interpolation
- Frequency: FFT-based frequency domain blending
- Stochastic: Random pattern-based merging
Advanced Spectral Merging
The Prism node offers frequency-domain merging:
- Spectral Blend: Magnitude-based frequency separation
- Frequency Bands: Different weights for low/high frequency components
- Magnitude Weighted: Stronger tensor gets more influence
- Adaptive Mix: Similarity-based adaptive merging
- Harmonic Merge: Phase relationship-based blending
Granular SDXL Block Control
SDXL Block Merge and SDXL Advanced Block Merge provide:
- Individual control over input blocks (0-11)
- Individual control over output blocks (0-11)
- Middle block control (3 components)
- Time embedding and label embedding control
- Final output layer control
- Memory-optimized processing for any GPU size
- Advanced memory management with context managers
- Automatic GPU/CPU fallback based on available memory
V-Pred/Epsilon Block Merging
The Epsilon/V-Pred Block Merge node provides:
- Automatic conversion between V-Prediction and Epsilon prediction types
- Granular control over individual blocks:
- Input blocks 0-8 (9 individual controls)
- Middle blocks 0-2 (3 individual controls)
- Output blocks 0-8 (9 individual controls)
- Final output layer control
- Support for mixed prediction type merging (e.g., Epsilon + V-Pred → Epsilon)
- Mathematical conversion factors for accurate type conversion
Pure Prediction Type Conversion
The Epsilon/V-Pred Converter provides:
- Pure conversion between prediction types without merging
- Configurable conversion strength (0.0 to 2.0)
- Smart layer targeting:
- Critical layers: Output blocks, middle blocks, final output
- Secondary layers: Time embedding, late input blocks
- Empirically derived conversion factors for mathematical accuracy
- Automatic model options updating for ComfyUI compatibility
Advanced CLIP Merging
The Advanced CLIP Merge node offers:
- Multiple Interpolation Methods:
- Linear: Standard weighted average
- SLERP: Spherical interpolation for vector-like parameters
- Cosine: Smooth cosine-based transitions
- Weighted Average: Magnitude-based automatic weighting
- Spectral Blend: Frequency domain blending with separate magnitude/phase control
- Layer-Specific Controls:
- Attention Bias: Adjust merging for attention layers
- Feedforward Bias: Control feedforward network blending
- Embedding Bias: Modify text/positional embedding merge ratios
- Normalization Bias: Adjust layer normalization merging
- Advanced Options:
- Preserve Norms: Maintain original parameter magnitudes
- Memory Efficient: Optimize for GPU memory usage
- Spectral Alpha: Control phase vs magnitude blending in spectral mode
Advanced Masking System
Create sophisticated merging patterns:
-
Generate Masks:
- Layer-based: Target specific model layers
- Block-based: Target transformer blocks
- Component-based: Target attention or feedforward layers
- Custom patterns: Use regex patterns
- Random sparse: Create random merging patterns
- Depth gradients: Gradual transitions through model depth
-
Filter and Blend Masks:
- Filter model keys by component type
- Combine masks with various blending modes
- Memory-efficient batch processing
-
Apply Masks:
- Use masks to control merge strength per parameter
- Selective merging based on model structure
- Tensor-level precision control
Model Weight Modification
Transform model weights directly:
- Multiply: Scale weights by a factor
- Add: Add constant values
- Set Value: Replace weights with specific values
- Clamp Magnitude: Limit weight magnitudes
- Scale Max Abs: Normalize based on reference model
Workflow Organization
Use the Checkpoint Reroute + Notes node to:
- Clean up complex workflows with multiple model connections
- Add documentation and notes directly in your workflow
- Maintain MODEL, CLIP, and VAE connections without modification
- Keep track of model versions and merge parameters
- Zero processing overhead - direct passthrough routing
- Organize workflow structure for better readability
Node Reference
| Node | Category | Purpose | |------|----------|---------| | Main Merge | Tensor Prism/Core | Advanced merging with multiple methods | | Prism | Tensor Prism/Core | Spectral frequency-domain merging | | SDXL Block Merge | Tensor_Prism/Merge | Basic granular SDXL merging | | SDXL Advanced Block Merge | Tensor_Prism/Merge | GPU-optimized SDXL merging | | Epsilon/V-Pred Converter | Tensor_Prism/Convert | Pure prediction type conversion | | Advanced CLIP Merge | Tensor_Prism/CLIP | Sophisticated CLIP merging with multiple methods | | Model Mask Generator | Tensor Prism/Mask | Create structural masks | | Weighted Mask Merge | Tensor Prism/Mask | Apply masks to merging | | Model Key Filter | Tensor_Prism/Mask | Filter model parameters | | Mask Blender | Tensor_Prism/Mask | Combine multiple masks | | Model Weight Modifier | Tensor_Prism/Transform | Direct weight manipulation | | Checkpoint Reroute + Notes | Tensor_Prism/Utilities | MODEL/CLIP/VAE rerouting with workflow documentation |
Memory Management
The TensorPrism pack includes advanced memory management features:
- Automatic GPU Detection: Optimizes for your specific GPU memory
- Adaptive Batch Sizes: Adjusts processing based on available memory
- Precision Selection: Automatic FP16/FP32 based on memory constraints
- Progressive Cleanup: Aggressive garbage collection for low-memory systems
- CPU Fallback: Automatic fallback when GPU memory is insufficient
- Memory Context Managers: Automatic cleanup and resource management
- Threshold-Based Processing: Memory usage monitoring with configurable limits
Recommended Settings by GPU:
- 24GB+ (RTX 4090, etc.): Use default settings, batch size 50+
- 12GB (RTX 4070 Ti, etc.): Set memory limit to 8GB, enable auto precision, batch size 30-50
- 8GB (RTX 4060 Ti, etc.): Set memory limit to 6GB, force CPU for large merges, batch size 10-30
- 6GB and below: Use CPU processing for best stability, enable aggressive cleanup
Requirements
- ComfyUI
- PyTorch >= 1.12.0
- NumPy >= 1.21.0
- psutil >= 5.8.0 (for memory management)
Tips and Best Practices
- Start Conservative: Begin with lower merge ratios (0.3-0.7) and adjust based on results
- Use SLERP for Dissimilar Models: When merging very different models, SLERP often produces better results
- Leverage Spectral Methods: Frequency domain merging can preserve details better than linear methods
- Use Masks for Precision: Create masks to merge only specific model components
- Memory Management: Monitor memory usage and adjust batch sizes for your hardware
- Experiment with Spectral Parameters: Different frequency biases can dramatically change results
- Layer-Selective Merging: Use depth gradients for smooth transitions through model layers
- Prediction Type Awareness: Use the Epsilon/V-Pred nodes when working with models of different prediction types
- CLIP Merging Strategy: Use spectral blend for CLIP when preserving text understanding is critical
- Conversion Strength: Start with 1.0 conversion strength and adjust if results seem over/under-converted
- Document Your Workflows: Use the Checkpoint Reroute + Notes node to keep track of your merging experiments
- Organize Complex Workflows: Use reroute nodes with notes to create clean, documented workflow structures
- Layer-Specific CLIP Control: Use attention/feedforward bias to fine-tune CLIP behavior for specific use cases
- Zero Overhead Documentation: The reroute node adds no processing time while providing workflow organization
Troubleshooting
- Memory Issues: Reduce batch sizes, lower memory limits, or enable CPU fallback
- Poor Results: Try different merging methods or adjust spectral parameters
- Compatibility: Ensure models are the same architecture (SDXL with SDXL, etc.)
- Slow Performance: Check if you're accidentally using CPU when GPU is available
- Artifacts: Try more conservative merge ratios or use SLERP for smoother blending
- Prediction Type Issues: Use the Epsilon/V-Pred nodes for automatic type conversion
- CLIP Problems: Use preserve_norms=True and lower merge ratios for CLIP stability
- Conversion Artifacts: Reduce conversion strength or use pure converter instead of block merge
Performance Tips
- Batch Size: Larger batches are more efficient but use more memory
- Precision Mode: FP16 saves memory but may affect quality on some operations
- Memory Cleanup: Enable aggressive cleanup for systems with limited RAM
- Device Selection: Let the system auto-detect optimal device unless you have specific needs
- Workflow Organization: Use reroute nodes to reduce visual complexity without performance impact
- CLIP Memory: CLIP merging is less memory-intensive than UNet merging
- Conversion vs Merging: Pure conversion uses less memory than block-level merge+convert
License
https://www.gnu.org/licenses/gpl-3.0.en.html
Contributing
Contributions are welcome! Please feel free to submit issues and pull requests.
Changelog
Version 1.2.0
- Added Advanced CLIP Merge node with multiple interpolation methods and layer-specific controls
- Added Epsilon/V-Pred Converter node for pure prediction type conversion
- Enhanced SDXL Advanced Block Merge with improved memory management and context managers
- Improved Weighted Mask Merge with tensor-level precision control
- Added spectral blending capabilities to CLIP merging
- Enhanced memory management with automatic GPU/CPU fallback
- Improved layer identification and targeting for prediction type conversion
- Better mathematical accuracy in conversion factors
- Added Epsilon/V-Pred Block Merge node with granular block control and prediction type conversion
- Added Checkpoint Reroute + Notes (Tensor Prism) node for workflow organization and documentation
- Enhanced block-level merging capabilities with individual layer control (input blocks 0-8, middle blocks 0-2, output blocks 0-8)
- Improved prediction type conversion with mathematical accuracy
- Better workflow documentation and organization features
- Zero-overhead utility nodes for complex workflow management
- Improved tooltip system and user experience enhancements
Version 1.1.0
- GPU-optimized memory management
- Cross-platform compatibility (CUDA/MPS/CPU)
- Bunch of Bug Fixes
- Addition of 3 Nodes
Version 1.0.0
-
Initial release
-
Core merging nodes with advanced interpolation methods
-
Advanced mask system with filtering and blending
-
Spectral analysis and frequency-domain merging
-
Advanced mask system with filtering and blending
-
Support for SDXL models with granular block control
-
Model weight modification tools