Glm_4v_9b

If you've ever wished your image captions read like a person describing a photo instead of a Danbooru tag dump, Glm_4v_9b is the node to try. It runs Zhipu AI's open glm-4v-9b vision model fully locally: you feed it an image and a question, it looks at the picture and answers in plain language. Where a WD14-style tagger outputs comma lists, this gives you a sentence - which is exactly the kind of input LLM-encoded image models and prompt-enhancement loops eat for breakfast. The catch is the price of admission: this model is heavy.

How it works

Same machinery as its text-only sibling Glm_4_9b_Chat, plus vision. Your IMAGE tensor gets converted to a PIL image, then AutoTokenizer.apply_chat_template() is called with the image attached to the user turn, and AutoModelForCausalLM runs the generation in bfloat16 on CUDA with do_sample=True and top_k. The vision encoder is glued into the model itself, so you don't need a separate captioning pipeline - it's one node, image in, text out.

The inputs that matter

• repo_id - forced input, meaning it must be wired, not typed. Feed it from the pack's Glm_Lcoal_Or_Repo selector (pick THUDM/glm-4v-9b) or any string holding a valid repo path.
• image - any IMAGE from your graph: a Load Image, a KSampler's decoded output, a VHS frame. The node is happy to caption things you just generated.
• user_content - your question or instruction. Default is "描述这张图片" (describe this image); swap in pointed prompts like "what's the lighting direction and lens?" for targeted analysis.
• max_length (default 2500) and top_k (default 1) - same as the chat node: length cap and sampling greediness.
• reply_language - appends a language instruction ("answer me in English", "用中文回复我", etc.) to your prompt.

Output: a STRING named prompt. Classic wiring: run it on a reference image, take the caption, and feed it into a CLIP text encode to re-generate in the same style - prompt-from-image without ever leaving the graph.

Installing it

It ships in the pack, so install once:

cd ComfyUI/custom_nodes
git clone https://github.com/smthemex/ComfyUI_ChatGLM_API

Restart, and the node shows up under ChatGlm_Api (search "chatglm"). No API key, no config.json - this is the local path, not the cloud one. Do update transformers first (pip install -U transformers); the pack's pinned 4.41.1 is too old for GLM-4 repos, a contradiction the README fesses up to.

Where people get burned

• The model is enormous. glm-4v-9b is roughly 26GB and needs north of 28GB VRAM in 16-bit. The community consensus is that you run it at 4-bit to get it under ~11GB - which this node does not do. If you're not on a 24GB+ card, this node will OOM before it says a word.
• First run downloads the whole model from HuggingFace. Budget the disk and the wait.
• A shipped-code quirk: the 4v node's decode line is commented out in the source and it tries to .strip() the raw tensor instead. If it errors with something like 'Tensor' object has no attribute 'strip', that's the author's bug, not your setup - you're stuck editing the node file or switching to a maintained GLM-4V loader.

For the API version of this same capability (small image, no local model, tiny cost), the pack's ZhipuaiApi_img node is the light option. This one is for people with VRAM to spare who want vision analysis that never leaves the machine.