ComfyUI-Lance

ComfyUI-Lance

Custom ComfyUI nodes for ByteDance Lance — a unified 3B multimodal model supporting image and video understanding, generation, and editing in a single framework.

A video I posted about unified multimodal, and this model : https://youtu.be/fvEeEW6trXA

Official sources

• Model weights: bytedance-research/Lance on Hugging Face
• Inference pipeline: bytedance/Lance (inference_lance.py, inference_lance.sh)

This pack bundles the upstream modeling/ code in lance_source/ and calls the same validate_on_fixed_batch path as the official CLI.

Features

| Node | Description | |------|-------------| | Lance Model Loader | Load and cache the Lance model (LLM + ViT + VAE) | | Lance Text-to-Image | Generate images from text prompts | | Lance Text-to-Video | Generate videos from text prompts | | Lance Image Edit | Edit images with text instructions | | Lance Video Edit | Edit videos with text instructions | | Lance Image Understanding | Ask questions about images (VQA) | | Lance Video Understanding | Ask questions about videos (VQA) |

<img width="1041" height="893" alt="imageedit" src="https://github.com/user-attachments/assets/0520c0d4-2579-440a-b0a9-d83adf375a36" /> <img width="1815" height="844" alt="Screenshot 2026-05-19 053929" src="https://github.com/user-attachments/assets/a49b5cfe-8e72-46de-bfb2-b75ddd0cfd51" /> <img width="1491" height="965" alt="Screenshot 2026-05-19 054507" src="https://github.com/user-attachments/assets/acbdb074-91a6-48c0-b484-82e4a8a58407" />

Installation

1. Clone into ComfyUI custom nodes

cd ComfyUI/custom_nodes/
git clone https://github.com/benjiyaya/ComfyUI-Lance.git

2. Install dependencies

pip install -r ComfyUI-Lance/requirements.txt

3. Download model weights

Download the full Hugging Face repo (or individual folders) into ComfyUI/models/lance/.

The HF layout uses Lance_3B / Lance_3B_Video; the loader also accepts lance_3b / lance_3b_video (same as upstream downloads/ in the GitHub repo).

ComfyUI/models/lance/
├── Lance_3B/               # Image tasks (HF name) — or lance_3b/
│   ├── llm_config.json
│   ├── model.safetensors
│   └── tokenizer files...
├── Lance_3B_Video/         # Video tasks (HF name) — or lance_3b_video/
│   ├── llm_config.json
│   ├── model.safetensors
│   └── tokenizer files...
├── Qwen2.5-VL-ViT/         # Vision encoder
│   ├── vit.safetensors
│   └── config.json
└── Wan2.2_VAE.pth          # VAE weights (repo root on HF)

Upstream places the same files under downloads/; ComfyUI uses models/lance/ via folder_paths.

The Lance model source code is bundled in lance_source/ — no separate clone needed.

Advanced: Set LANCE_REPO_PATH env var to use a custom Lance repo instead of the bundled source.

Hardware Requirements

• GPU: NVIDIA GPU with ≥40GB VRAM (e.g., A100 40GB, A6000)
• CUDA: 12.4+
• Python: 3.10+

Usage

Basic Text-to-Video

[Lance Model Loader] → [Lance Text-to-Video] → [ComfyUI Save Video]

1. Load the model with lance_3b_video variant
2. Enter your prompt in the Text-to-Video node
3. Set resolution, frames, CFG, seed
4. Connect to ComfyUI's native Save Video to export as MP4

Image Understanding

[Load Image] → [Lance Image Understanding]

1. Load the model with lance_3b or lance_3b_video
2. Connect an image and type your question
3. Output is a text string

Video Edit / Video Understanding

[Load Video] → [Lance Video Edit] / [Lance Video Understanding]

Connect frame batches from ComfyUI's native Load Video node or VideoHelperSuite's Load Video. The node converts frames to a temp video file for Lance processing.

Key Parameters

Aligned with official inference defaults:

| Parameter | Default | Description | |-----------|---------|-------------| | num_timesteps | 30 | Denoising steps (30 or 50 recommended) | | timestep_shift | 3.5 | Flow matching schedule shift | | cfg_text_scale | 4.0 | Classifier-free guidance scale | | num_frames | 50 | Video frames (max 121); T2I uses 1 | | resolution | video_480p / image_768res | Preset per task |

Task ↔ model variant (official)

| Task | ComfyUI node | Loader variant | resolution | Notes | |------|--------------|----------------|--------------|-------| | t2i | Text-to-Image | lance_3b | image_768res | text_template=False, num_frames=1 | | t2v | Text-to-Video | lance_3b_video | video_* preset | text_template=True, frame count snapped to 4n+1 | | image_edit | Image Edit | lance_3b | image_768res | text_template=True, size from input | | video_edit | Video Edit | lance_3b_video | video_480p | text_template=True, size from input | | x2t_image | Image Understanding | either | — | No diffusion steps | | x2t_video | Video Understanding | lance_3b_video | — | No diffusion steps |

Model Caching

The model is loaded once and cached per GPU. Multiple nodes can share the same loaded model without re-loading.

License

This ComfyUI wrapper is provided as-is. The Lance model is licensed under Apache 2.0 by ByteDance. See https://github.com/bytedance/Lance for details.