Rex-Omni Detector

You know the feeling: you need to find the person in an image, so you wire up a detector, then a different detector for the text, then a pose model for the keypoints. Rex-Omni Detector is the "just do it" node - one multimodal model, eight tasks, one image in. It's not a diffusion node at all. It's a vision-language model that reads your image and answers with coordinates, which makes it genuinely useful for auto-labeling, grounding regions for inpainting or ControlNet-style workflows, and pulling text out of screenshots.

What it does

The Detector takes an image and a task, runs the model loaded by the Rex-Omni Loader, and hands back annotated boxes, points, text, and keypoints. The task dropdown gives you eight modes:

• detection - plain object detection with bounding boxes
• pointing - a single click point on the target
• visual_prompting - find everything similar to a reference box
• keypoint - person or animal skeleton
• ocr_box and ocr_polygon - text detection, box or polygon
• gui_grounding and gui_pointing - UI elements in box or point form

How it works

Under the hood it's text generation, not a classic YOLO head. The node converts your ComfyUI tensor to a PIL image, builds a prompt from the task and your text, and runs a Qwen2.5-VL-based model. The model answers in special tokens like <|box_start|><0><35><980><987><|box_end|>, which a parser converts from 0–999 bin coordinates back into real pixel coordinates. It then draws the results - boxes, points, or keypoint skeletons with limb connections - onto a visualization image. This is why it feels slower than a dedicated detector: every image is a full forward pass through a 7B-class VLM. The trade is that one model covers detection, OCR, pose, and GUI grounding, where you'd otherwise stack four packs.

The inputs that matter

• task - pick your mode from the eight above.
• text_prompt - what to look for. Defaults to a person, but it's task-aware: for detection type the category ("person", "car"), for OCR type "text" or "words", for GUI tasks type the element name like "button".
• keypoint_type - only appears for the keypoint task: person or animal. With animal, your text_prompt becomes the category list.

One honest warning: visual_prompting is in the dropdown, but the node never exposes an input for the reference boxes the task requires. Pick it and you'll get a "Visual prompt boxes are required" failure straight from the wrapper. The other seven modes work; treat that eighth as a placeholder.

Outputs

The node has six, which is a lot until you realize half of them are the same data in different wrappers:

• visualization (IMAGE) - the annotated image; wire it to a Preview or Save node.
• result_text (STRING) and predictions_json (STRING) - human-readable and structured results, ready for any text node or a JSON parser.
• bboxes (BBOX) - the extracted boxes in a type that box-aware packs, like Impact Pack's region tooling, know how to read.
• texts (STRING) - OCR text content.
• keypoints (STRING) - keypoint coordinates.

Install

The pack is one clone and a big model download:

cd ComfyUI/custom_nodes/
git clone https://github.com/flybirdxx/ComfyUI-RexOmni.git

Restart ComfyUI, then grab the model (nothing is auto-downloaded):

pip install huggingface_hub
huggingface-cli download IDEA-Research/Rex-Omni --local-dir models/Rex-Omni

Run that from the ComfyUI root. The full install story - including the qwen-vl-utils dependency the requirements.txt forgets to list, and the flash-attention gotcha - is on the Rex-Omni Loader page, and it applies to you too.

Troubleshooting

• "Rex-Omni 模块未正确安装" - the wrapper imports failed; almost always missing qwen-vl-utils. pip install qwen-vl-utils and restart.
• Task errors - if visual_prompting fails, that's the known limitation above, not your fault.
• Garbage or missing results - check your text_prompt; the model only finds what you name. "a person" in a crowd scene will return a few boxes, not fifty.
• The failure path is rough. When the detector errors internally it returns a Chinese string like 检测失败: ... as result_text, and the exception handler in the bundled code has a tuple bug that can crash instead of returning. It's a Claude-written pack, as the README candidly admits - rough edges included. For serious auto-labeling pipelines you might still reach for dedicated tools, but for a do-it-all vision node in one graph, this is the one to grab.