Omost LLM HTTP Server

If Omost LLM Chat is the engine, this is the turbo button. Omost LLM HTTP Server loads no model at all - it wraps a client for an LLM you're already running elsewhere, so layout generation happens outside ComfyUI and your diffusion VRAM stays untouched. It was added in June 2024 specifically because the local path is so slow, and it's the node most long-term Omost users end up living on.

Two inputs:

• address - the base URL of your LLM server.
• api_type - OpenAI or TGI.

The mechanism is almost boring. For an OpenAI-compatible endpoint it appends /v1 to your address (unless you already included it) and talks to /v1/chat/completions. For TGI it does the same, but first hits the server's /info endpoint to discover the model_id, because TGI wants one in the request. Either way you get an OMOST_LLM output that plugs straight into Omost LLM Chat - same downstream behavior, no code changes, and it still works with the multi-round conversation loop.

Why bother? The README is blunt about it: in-process inference takes 3–5 minutes per chat on a 4090. Deploy the same 8B model under TGI in Docker and you get up to 6x faster. Run the Q8_0 GGUF through llama.cpp's llama-server and the README's author measured 30–40 seconds per image, versus 50–60 seconds with llama-cpp-python on the same 3090 Ti. The trade is roughly 20GB of VRAM for an 8B TGI deployment, plus a container you have to keep alive. For serious work, it's worth it.

The README's TGI recipe:

docker run --gpus all -p 8080:80 \
  -v $HOME/.cache/huggingface/hub:/data \
  ghcr.io/huggingface/text-generation-inference:2.0.4 \
  --model-id lllyasviel/omost-llama-3-8b --max-total-tokens 9216 --cuda-memory-fraction 0.9

Test it with curl on the port before wiring up the node:

curl 127.0.0.1:8080/generate -X POST \
  -d '{"inputs":"What is Deep Omost?","parameters":{"max_new_tokens":20}}' \
  -H 'Content-Type: application/json'

Gotchas that actually bite: the address must be reachable from the machine running ComfyUI - 127.0.0.1 only works if the server is on the same box. The model behind the endpoint should be one of the Omost-tuned ones; a generic LLM won't emit the Canvas code the chat node requires, and you'll get a "Response does not contain codes!" error. And pick api_type to match reality: TGI only for a real TGI instance, OpenAI for llama.cpp's OpenAI-compatible server and everything else.