Omost LLM Loader

Omost LLM Loader is the self-contained way to get the brains behind Omost running inside your ComfyUI process. One input, one output: pick a model from the llm_name dropdown and you get an OMOST_LLM to feed into Omost LLM Chat. It exists because lllyasviel's Omost needs a small LLM to compose layouts, and this node is how you run that model without leaving ComfyUI.

The dropdown offers the three official Omost checkpoints:

• lllyasviel/omost-llama-3-8b-4bits (default)
• lllyasviel/omost-phi-3-mini-128k-8bits
• lllyasviel/omost-dolphin-2.9-llama3-8b-4bits

Mechanically it's plain transformers under the hood: AutoModelForCausalLM.from_pretrained with device_map="auto" and trust_remote_code=True, so the model is pulled from HuggingFace on first use and offloaded across CPU/GPU as needed. The "4bits"/"8bits" in the names means the download is already quantized - llama-3-8b-4bits is around 5GB - and loading it needs bitsandbytes, which is why the pack's requirements.txt pins bitsandbytes>=0.43.1 alongside transformers>=4.41.1. The model file lives in your HuggingFace cache, not in ComfyUI/models, so nothing to download through ComfyUI's model browser.

The catch, and it's the same one every Omost guide mentions: the official in-process method is slow. Each chat takes about 3–5 minutes on a 4090, because you're doing a full autoregressive generation of a long structured layout through a small quantized model. Fine for a one-off, miserable for iterating. The README's recommended fix is to skip this node entirely and use Omost LLM HTTP Server against TGI (up to 6x faster, at the cost of ~20GB of VRAM for an 8B deployment) or a llama.cpp server running the Q8_0 GGUF (roughly 30–40s per image on a 3090 Ti, per the README's measurements).

If you do run it locally: the first load downloads the model, so expect a wait, and the model shares your VRAM with the diffusion model since it's all in one process. On a 12–16GB card you may need to generate in a separate pass or you'll be swapping. If the nodes show up red in your graph instead of loading, that's almost always a missing dependency - install the pack's requirements (transformers, bitsandbytes, protobuf) via ComfyUI Manager's "Install missing custom nodes" prompt or pip directly.