M8 · 本地大模型推理

The standing request in this community is to run a GGUF chat model directly as a Comfy node without a separate Ollama server sitting in the middle. M8 · Local LLM Inference is that, and the name is not a lie: it loads a .gguf off your disk with llama-cpp-python, needs no API key, and never touches the network. The cost is the same one GGUF always charges - you're now budgeting VRAM for two models at once instead of one.

Why local, when the API node exists

Not quality. An 8B local model is not out-writing a frontier endpoint. What it buys you is being uncensored, offline, and free per call - which is exactly what you want for the boring high-volume jobs: rewriting a rough idea into a structured prompt, or reshaping a caption before it goes to the encoder. It's also one process instead of two, which matters more than people expect when you're debugging a workflow at 2am.

How it works

Models live in models/LLM, and the node registers that folder with ComfyUI's folder_paths on load - so extra_model_paths.yaml still works and nothing is hardcoded to a drive letter. The main dropdown lists .gguf files with mmproj* files filtered out; the mmproj gets its own dropdown and is auto-paired by name, which you override when the guess lands wrong.

Loaded models are cached in memory, keyed on model path, mmproj, context length and GPU layers. That's not a nicety - a GGUF load takes anywhere from one to ten-plus seconds, and reloading it every run would make the node useless. Switching models drops the old one, so you're not stacking weights.

Messages are assembled the OpenAI way: the system block is skill first, then system_prompt; the user block is user_prompt with extra_text appended. Images go in as JPEG data URLs, resized so the long side stays at or under 1024 px. That resize is load-bearing, not cosmetic: vision encoding has a fixed token budget, so a full-resolution photo just eats context without adding detail. Think blocks are stripped out with a regex plus a truncation fallback, because a chatty local model will otherwise hand you its inner monologue instead of an answer.

The inputs and the two outputs

Required: model, system_prompt, user_prompt, extra_text. Optional: image, skill, mmproj, max_tokens (512), temperature (0.7), top_p (0.95), ctx (4096), gpu_layers (−1), thinking.

The three worth understanding: gpu_layers at −1 offloads as much as the GPU allows, which only helps if your llama-cpp-python was built with CUDA - 0 is CPU-only. ctx larger means more memory, and 4096 gets tight fast once you start feeding images. thinking is just a request flag, and some models ignore it entirely; if yours does, ask in the prompt too.

Outputs are text and status. Don't skip status - it's a one-line diagnostic like 2.8s · 190 tokens out · 61.2 tok/s · text only · reused cache · GPU available, and it's the fastest way to discover you're actually running on CPU. Wire it into a text display node if you'd rather not open the console.

Installing it (the only real work in this pack)

The pack itself has zero dependencies, and the local node is deliberately opt-in:

cd ComfyUI/custom_nodes
git clone https://github.com/chenr5934-tech/ComfyUI-M8nodes.git

Then, for this node only:

python -m pip install llama-cpp-python

Install it into the same Python ComfyUI runs with - on a portable build that's python_embed, not your system Python. The standalone build is where people get stuck: llama-cpp-python doesn't ship official wheels for every Python version ComfyUI packages, so a plain pip install can end up compiling from source. If you hit that wall, prebuilt CUDA wheels circulated by the community are the usual escape hatch - the filename encodes what you need (cp313 = Python 3.13, cu130 = CUDA 13.0), and you install the .whl directly with python -m pip install <wheel>.

If it's missing or broken, the node reports M8-LLM-015 at run time instead of the pack failing to load - every other node keeps working.

Where it bites

M8-LLM-016 with a vision model means your llama-cpp-python is too old for that mmproj → pip install -U llama-cpp-python. M8-LLM-020 on inference failure points at the two usual causes: context too small (raise ctx) or the GPU can't fit it (set gpu_layers to 0).

Vision with no mmproj selected does nothing at all - no error, just text-only answers. Check that dropdown before you blame the model. And the VRAM maths is real: a Q4_K_M quant in the 3B–8B range with a modest ctx is the sane pairing if the diffusion model has to live on the same card.