LLM Server Embeddings (GGUF)

If you run SillyTavern against a local model, you've probably wanted Vector Storage - the long-term memory feature that embeds your chat history and retrieves the relevant bits. Normally that needs a second endpoint: a separate embedding server, another port to configure, another process to keep alive. This node puts the embedding model on the same address as your chat model.

It's a config node. It generates nothing; it emits one bundle you wire into Local LLM Server's embeddings input, and it's one of three optional side models in that package (draft, vision, embeddings), each with its own node so the settings only exist when you use them.

What the server side is doing

The llm_server package runs llama-server or koboldcpp under ComfyUI behind a small gateway that stays listening for as long as ComfyUI runs, while the models behind it are killed for an image render and reloaded for the next message. That's what makes a chat client and a diffusion model share one card without SillyTavern ever seeing a closed port.

How the embedder fits depends on the backend, and the difference isn't cosmetic. On llama-server, --embeddings restricts a process to embedding work, so the embed_model gets a second, embedding-only process of its own and the gateway routes /v1/embeddings and /rerank to it while chat goes to the other. On koboldcpp it loads inside the one process. Either way the embedder obeys the same unload rules as the chat model and is loaded on demand by the first request that needs it - so ask for a vector with nothing loaded and only the embedding model wakes up, not the whole chat model.

The inputs

embed_model is the embedding .gguf - EmbeddingGemma, nomic-embed, bge, whatever you have - listed from ComfyUI/models/llm. embed_model_path is the manual path when the dropdown is on its placeholder. And to be clear, because this is the mistake: it is not a chat model. Feeding it a 7B instruct model that happens to be in the same folder gets you an embedding-only process that can't chat and a chat model that can't embed.

embed_n_ctx = 0 means the model's trained default, which is what you want unless it complains. embed_n_gpu_layers = -1 is all layers on the GPU; these models are small, so that's normally right, and 0 keeps it on the CPU (koboldcpp only distinguishes 0 from not-0, so any positive number means "GPU" there).

pooling is the one setting that can quietly ruin your day: it decides how the per-token vectors become one vector - model default, none, mean, cls, last, rank. Leave it on the model's default unless the model card says otherwise. The wrong pooling doesn't throw an error; it gives you vectors that are consistently, uselessly shaped, and retrieval quality just gets worse. It's llama-server only.

rerank exposes /rerank for reranker models - meaningless with a plain embedder, so only turn it on if you know what you're serving. embed_extra_args is raw flags for the embedding process alone, e.g. --embd-normalize 2, and they don't touch the chat process.

Output: embeddings (KINBURG_LLM_SERVER_EMBED), which goes into the server node's embeddings slot. The three side-model types are distinct so you can't plug the vision config into this one by accident.

Install

ComfyUI Manager (search "Kinburg"), or:

cd ComfyUI/custom_nodes
git clone https://github.com/Kinburg/Kinburg-Nodes
# restart ComfyUI

Manager runs the pack's install.py, which installs the CUDA-matched llama-cpp-python wheel for you (0.3.30, matched to your torch's CUDA major - a mismatch is the "llama.dll or one of its dependencies is missing" failure). The server binary isn't bundled: download llama-server or koboldcpp yourself and put it under ComfyUI/models/llm, where the node's binary dropdown can see it. Embedding GGUFs go in the same folder; the model dropdown walks it recursively, so subfolders show up as nested entries.

Gotchas

pooling and rerank are llama-server only. Set them with koboldcpp and nothing happens - the server node reports what it ignored in its status output, which is the honest way to find out rather than guessing.

Vectors are per-model. If you embed a chat log with nomic-embed and then swap to bge, everything stored before is garbage relative to everything after. Reindex, or keep the model.

Embedding models eat context linearly. embed_n_ctx sets the ceiling, and SillyTavern chunks what it stores, so a small context isn't fatal - but a chunk that gets truncated mid-sentence embeds as a half-idea, and that's what gets retrieved later.