LaVi-Bridge Model Loader

LaVi-Bridge is a 2024 research idea: instead of CLIP trying to parse your prompt, let a real language model (T5 or Llama-2) read it, then patch a small adapter and a couple of LoRAs over an ordinary SD1.5 checkpoint. The whole thing only ships LoRAs and adapters - it never retrains the diffusion model. Kijai wrapped it for ComfyUI, and this is the node at the start of every workflow: it takes your checkpoint, converts it into the diffusers machinery the bridge needs, and staples on the visual LoRA.

Here's the thing that trips people up: the wrapper doesn't come with a model. LaVi-Bridge assumes you already have an SD1.5 checkpoint, and the Model Loader is how you hand it over. Wire a CheckpointLoaderSimple into model and vae, pick a lora_type, and it does the rest.

What it actually does

The mechanism is worth knowing because it explains the whole "don't feed it SDXL" rule. The node pulls the state dict out of your ComfyUI model, converts it into a diffusers UNet2DConditionModel plus AutoencoderKL using a hardcoded v1-inference.yaml config - the SD1.5 architecture file. Then it monkeypatches the LaVi-Bridge visual LoRA (lora_vis.pt, rank 32) onto the UNet's ResNet blocks and attention layers. Result is cached, and it only re-runs if your model, VAE, or lora_type changes.

Because the conversion is pinned to the v1 inference config, this is strictly an SD1.5 toy. The community confirmed it back in 2024: LaVi-Bridge has no SDXL support, so don't point this node at an XL checkpoint and expect magic. Any SD1.5 model works - people ran Realistic Vision through the original test script fine.

The inputs that matter

• model (MODEL) and vae (VAE) - straight out of a standard CheckpointLoaderSimple. That's the only "real" model you supply; everything else is autodownloaded.
• lora_type - a dropdown with t5_unet (default) and llama2_unet. This must match the encoder node you use: t5_unet with the LaVi-Bridge T5 Encoder, llama2_unet with the LLaMA one. Pick T5. The README is blunt that the Llama-2 version "couldn't get anything useful" out of it, so the default is also the right answer.

It returns a single lavibridge output (type LAVIBRIDGE) that feeds straight into the LaVi-Bridge Sampler. Nothing else plugs into it - the sampler is the only consumer.

Installing it

Same story as any Kijai pack: install from ComfyUI Manager by searching "ComfyUI-LaVi-Bridge-Wrapper", or

cd ComfyUI/custom_nodes
git clone https://github.com/kijai/ComfyUI-LaVi-Bridge-Wrapper
cd ComfyUI-LaVi-Bridge-Wrapper
pip install -r requirements.txt

Portable installs should run python_embeded\python.exe -m pip install -r ... from the ComfyUI_windows_portable folder. The requirements file is thin - diffusers>=0.26.0, sentencepiece, peft>=0.8.2 - but the code also imports transformers and omegaconf without listing them, so if you hit an ImportError on either, pip install transformers omegaconf and move on.

On first run it downloads the bridge weights from shihaozhao/LaVi-Bridge into ComfyUI/models/lavibridge/ automatically. No model browsing, no Manager step - just let it run once and wait.

Common issues

The biggest practical trap is patience: first load triggers a Hugging Face download plus a checkpoint conversion, so the node can sit there for a couple of minutes before it does anything. Don't assume it's frozen. If you switched lora_type but the encoder node doesn't match, you'll get garbage or a type mismatch - keep both on t5_unet. And remember this wrapper is a fossil from the early-2024 "swap CLIP for an LLM" wave (the same idea as ELLA); nobody's actively maintaining it, so treat it as a working experiment, not a supported product.