LLaSM Interface

ComfyUI is famously good at making pictures, but this node is the one that makes it listen. You feed it a recording plus nothing else, and it answers in natural language - transcribing what was said, summarizing it, or fielding questions about the audio. It's the inference heart of the LLaSM pack: the other two nodes exist purely to feed it.

What LLaSM actually is

LLaSM ("Large Language and Speech Model") comes from LinkSoul, a Chinese open-source AI community. Architecturally it's LLaVA's trick applied to sound instead of pixels: a Whisper-large-v2 encoder converts the audio into features, a linear projector maps those into the token space of a 7B LLM (Llama-2 or Baichuan), and the LLM does the talking. This node is the ComfyUI wrapper around that final generation call.

How it works

Peek at the source and the flow is straightforward. Your audio arrives as an AUDIO value, and a WhisperProcessor turns the waveform into mel-spectrogram input features. Meanwhile the node builds a conversation where the "human" turn is just a run of special placeholder tokens - <au_start>, a bunch of <au_patch> tokens, <au_end> - so the LLM knows "audio is coming." Then model.generate() runs with those features injected, and whatever comes back is decoded to a plain STRING on the output pin.

The inputs that matter

Three required: audio (AUDIO, from LLaSM Load Audio or anything else emitting that type), llasm_model (the LLASM2MODEL from the loader), and prompt. Here's the honest gotcha: in the current build the prompt field is decorative. The shipped code declares it but never actually injects it into the conversation - the human turn is built from audio placeholder tokens only. So don't expect "summarize this in bullet points" to steer anything yet; the model just hears the audio and responds however its system prompt ("You are a helpful language and speech assistant…") moves it.

The optional knobs, quickly:

• keep_model_loaded - default False, which means the node shoves the whole model off the GPU and clears VRAM after every run. Your next run pays a real reload tax. Flip it on when you're iterating.
• max_new_tokens - caps the response length (default 2048, ceiling 4096). Lower it for quick test runs.
• temperature, do_sample, seed - the usual sampling trio. The 0.2 default is fairly greedy, which suits transcription.
• audio_token_len - how many <au_patch> tokens to reserve for the clip (default 64). Leave it alone unless long audio feels truncated.
• sampling_rate - leave at 16000; that's what Whisper wants.

The single output, output (STRING), wires into any text-display or save-text node.

What you need underneath

This is a real 7B LLM running fp16 plus a ~3GB Whisper tower. Call it 13–14GB of weights before activations - a 16GB card is uncomfortable, 24GB is where it stops being a fight. And the first run downloads everything into ComfyUI/models/LLM/, which looks like a freeze. It isn't; give it time.

One more honest note: this node has essentially zero community footprint - the page you're on shows a couple of stray impressions. It's an experiment for people curious about speech-LM ComfyUI, not a battle-tested workflow. If it errors, suspect the Model Loader first; the whole pipeline is downstream of it.