Extensions/Shrug-Prompter: Unified VLM Integration for ComfyUI
ComfyUI Extension

Shrug-Prompter: Unified VLM Integration for ComfyUI

A comprehensive Vision-Language Model (VLM) integration system for ComfyUI with more intelligent prompt optimization, object detection, template support, and performance optimizations. Optimized for Wan2.1, Flux Kontext, and general purpose. Goes well with my other project, an MLX/llama.cpp server with hot swappable models and ollama api compatibility, (heylookitsanllm)a/https://github.com/fblissjr/heylookitsanllm

By fblissjr·Created about a year ago·Updated 4 months ago· 25
fblissjr/shrug-prompter
Nodes10
On cloudLocal install
Categoryshrug
Stars25
Updated4 months ago

Nodes (10)

Shrug Accumulator

Per-graph memory for loops that need to remember things

shrug
Shrug ASR

Turn audio inside your graph into text you can prompt with

shrug
Shrug Cache Control

Peek inside (or flush) your server's KV cache

shrug
Shrug Connection

The one node that makes the whole pack work

shrug
Shrug Embeddings

Real vectors out of your local server, for search and similarity

shrug
Shrug Frame Pair

The start/end frame picker for interpolation loops

shrug
Shrug Template

255 ready-made prompts, served from a dropdown

shrug
Shrug Text Cleanup

The 30-second fix for ugly VLM output

shrug
Shrug VLM

A local Qwen3 brain in your graph, no API key required

shrug
Shrug VLM Batch

N prompts in one request instead of N slow round-trips

shrug
Readme

shrug-prompter

ComfyUI custom nodes for heylookitsanllm — a single-user MLX inference server on macOS.

Ten nodes, ~1,500 lines of Python. No multi-provider abstraction; this package is specifically tuned for heylookitsanllm's API surface (multipart vision upload, batch endpoint, Qwen3 thinking blocks, KV-cache management).

Install

Clone into ComfyUI/custom_nodes/:

cd ComfyUI/custom_nodes
git clone <this repo> shrug-prompter

Dependencies are declared in pyproject.toml and are normally already present in a ComfyUI environment. If not:

cd shrug-prompter
uv sync

Nodes

All nodes live in the shrug category.

  • Shrug Connection — base URL, model, optional auth. Builds a SHRUG_CONN object every other node consumes.
  • Shrug VLM — the main chat/vision node. Optional image batch, all sampling params, Qwen3 thinking output, base64 or multipart transport.
  • Shrug VLM Batch — N prompts (+ optional N images) in one call via /v1/batch/chat/completions. 2-4× faster than N sequential calls.
  • Shrug ASR — speech-to-text from a ComfyUI AUDIO input.
  • Shrug Embeddings — text → tensor via /v1/embeddings.
  • Shrug Template — loads a .md file from templates/ as a prompt. Ships with 250+ style/system-prompt snippets in templates/styles/ plus a few VLM templates in templates/vlm/. See templates/README.md for frontmatter format.
  • Shrug Text Cleanup — none / basic / standard (NFC + collapse blanks) / strict (ASCII-only).
  • Shrug Accumulator — graph-scoped list state (append/reset/read/pick), for loop workflows.
  • Shrug Frame Pair — frame-pair extractor for video interpolation.
  • Shrug Cache Control — list or clear the server's KV cache.

Configuration

Shrug Connection:

  • base_url — e.g. http://localhost:8080 (same-host) or https://your-llm-host (LAN). Caddy does TLS on the LAN; any reverse proxy will do.
  • model — the model widget is populated by JS fetching /v1/models on the server. If the fetch fails, the widget falls back to free text.
  • api_key (optional) — sent as Authorization: Bearer <key>. Ignored by vanilla heylookitsanllm; useful for other OpenAI-compatible targets.
  • admin_token (optional) — sent as X-Heylook-Admin-Token on admin endpoints only (currently just /v1/cache/clear). Matches the server's HEYLOOK_ADMIN_TOKEN env.

Testing

uv run pytest tests/ -v

67 tests covering the client (respx-mocked), media codecs, node behavior, and template parsing.

Live smoke test against a running server:

uv run python scripts/smoke.py --base-url https://your-llm-host --model your-model-id

Development

See CLAUDE.md for architecture notes and conventions.

Repo layout:

shrug-prompter/
├── __init__.py              # V3 API probe + /shrug/get_models route
├── client.py                # async httpx client
├── media.py                 # tensor <-> bytes (image, audio)
├── nodes.py                 # 10 ComfyNode classes + extension
├── pyproject.toml
├── templates/               # .md prompt library
├── web/provider.js          # ShrugConnection's model-dropdown widget
├── scripts/smoke.py         # live LAN smoke test
└── tests/                   # pytest

License

MIT.