LLaSM Model Loader

The LLaSM Model Loader is the heavy-lifting front end of this pack. It picks which LLaSM speech-language model you want, pulls it (plus the audio encoder) from Hugging Face if it isn't local yet, loads everything into VRAM, and hands the assembled bundle downstream as a single LLASM2MODEL. In practice you run this node once, then the LLaSM Interface node does the actual work.

What you're choosing

Two dropdown options on llasm_model: LinkSoul/LLaSM-Cllama2 (the default, Llama-2-based) and LinkSoul/LLaSM-Baichuan (Baichuan-based). Both are 7B-class models. Given LLaSM's Chinese origins, the Baichuan flavor sits more comfortably with Chinese speech; the Llama-2 one is the safer default for English. The third field, llasm_audio_tower, currently offers exactly one choice - openai/whisper-large-v2 - so there's nothing to decide there.

How it works

Straight from the source: on first run it snapshot_downloads the model repo into ComfyUI/models/LLM/<model-name>/. If that folder already exists it skips the download, which means deleting it is how you re-trigger one. Then the tokenizer loads, three special tokens get added (<au_patch>, <au_start>, <au_end>), the LLM loads in fp16 onto your GPU, and the Whisper processor and encoder load fp16 too. The final move is the LLaVA-style trick: the Whisper encoder gets spliced into the model's audio_tower slot so audio features flow straight into the LLM. All of it comes out the single llasm_model output (type LLASM2MODEL), which wires into the Interface node's llasm_model input.

The gotchas are real

First, the download. You're pulling roughly 13–14GB of fp16 weights for the 7B plus ~3GB for Whisper, straight off Hugging Face. The first run will look frozen; it isn't - go make coffee. The code even ships a commented-out HF_ENDPOINT line pointing at the China mirror (hf-mirror.com), which tells you exactly who this pack is built for and who's most likely to need that tweak.

Second, VRAM. 7B fp16 is ~13–14GB of weights before activations, and you also need room for Whisper. On an 8–12GB card this simply won't load; 16GB is marginal; 24GB is where it stops being a fight. This is the same wall the troubleshooting knowledge base describes for running any big transformer locally - the model doesn't shrink to fit.

Third, dependencies. The pack's requirements pin protobuf==3.19.0, an old and odd pin that can clash with other nodes in a shared environment. If protobuf errors show up after installing this pack, that pin is the usual suspect.

Installing it

Same steps as the rest of the pack: ComfyUI Manager, search "ComfyUI_LLaSM", or:

cd ComfyUI/custom_nodes
git clone https://github.com/leeguandong/ComfyUI_LLaSM

Then restart ComfyUI and let it resolve the requirements (transformers>=4.38, sentencepiece, librosa, accelerate, and the protobuf pin). After that, add this node to the graph, hit run once, and wait out the download before you blame anything else in the workflow.