Rex-Omni Loader

The Detector is the fun node. The Loader is where your weekend goes. Rex-Omni Loader is the node that grabs IDEA-Research's Rex-Omni model - a Qwen2.5-VL-based vision-language model that does detection, keypoints, OCR, pointing, and GUI grounding - loads it into VRAM, and hands it to the Rex-Omni Detector on a custom REX_OMNI_MODEL wire. There's exactly one model name in its dropdown ("Rex-Omni"), one output, and nothing else clever about it. All the cleverness - and all the pain - is in what it takes to get that model into memory.

How it works

On the first run, the Loader instantiates a wrapper around a Qwen2.5-VL model from the models/Rex-Omni/ directory. Under the hood it's plain transformers: Qwen2_5_VLForConditionalGeneration.from_pretrained in bfloat16 with device_map="auto", which means it'll happily eat whatever VRAM you have and spill the rest to system memory. Pick the vllm backend instead and it uses vLLM's LLM class, which defaults to grabbing 80% of GPU memory. Either way, this is a 7B-class vision model - bf16 weights alone run 14–16GB, so budget for a 24GB card before you get cozy.

The inputs that actually matter

Only two of the four inputs are worth touching on day one:

• backend - transformers (default) or vllm. Transformers just works if you have the deps; vllm is faster at batch inference but requires you to install vllm yourself, which is not in the pack's requirements.txt.
• max_tokens - 1–4096, default 1024. Bump it up if long OCR text keeps getting cut off.
• temperature - 0–2, default 0, meaning greedy decoding. For detection tasks you basically never want this above 0; it's there for captioning-style output.
• model_name - "Rex-Omni", always. The enum is hardcoded from the directory name, so there's no real choice.

The cache trap

Here's the thing that bites everyone: the Loader caches the loaded model by name and hands it back on every subsequent run. Change backend, max_tokens, or temperature after the first execution and nothing happens - it silently returns the already-loaded model with your old settings. The only way to apply new settings is to delete the Loader node and add a fresh one (or restart ComfyUI). If your vLLM backend "isn't working" after you switched from transformers, that's probably why.

Install

The README's official path is a plain clone:

cd ComfyUI/custom_nodes/
git clone https://github.com/flybirdxx/ComfyUI-RexOmni.git

Restart ComfyUI. ComfyUI Manager can usually find it by searching "Rex-Omni" as well.

Then the part the README is loudest about, because nothing is auto-downloaded: the model.

pip install huggingface_hub
huggingface-cli download IDEA-Research/Rex-Omni --local-dir models/Rex-Omni

Run that from your ComfyUI root so it lands in models/Rex-Omni/. It's a multi-GB pull; a "model not found" error almost always means you pointed this at the wrong folder or skipped it entirely.

Common issues

• "Rex-Omni 模块未正确安装" - the wrapper failed to import. The pack's requirements.txt is a wall of heavy deps (transformers, torchvision, opencv-python, pycocotools, scipy, shapely, timm, ninja...), but the one thing the code actually imports that isn't listed is qwen-vl-utils. Without it, the module never loads. pip install qwen-vl-utils fixes most "Rex-Omni 模块未正确安装" cases.
• Flash-attention errors - the transformers backend requests attn_implementation="flash_attention_2" by default. If your environment lacks flash-attn, loading can fail; you may need to install it (a CUDA build, bring coffee) or wait for a fallback path.
• Nothing happens after changing settings - see the cache trap above.
• Temper your expectations overall. The author is upfront in the README that this pack was written by Claude ("Since I'm not very familiar with coding") and that some features work while others still need fixing. It's a one-person research-model wrapper with essentially zero community footprint on reddit. The issues page on GitHub is your support channel, not a tutorial.

For what to do with the loaded model, head to the Rex-Omni Detector.