OmniParser_Loader

OmniParser is Microsoft's "pure vision based GUI agent" screen parser - the thing that turns a screenshot into a structured list of numbered text boxes and icons an LLM can actually act on ("click the button at box 7"). This node is the boring half of that pipeline: it doesn't parse anything, it loads the two models the Sampler needs and hands them to you as one bundle. But boring here means "does the heavy lifting once," so it's worth understanding before you wire up the rest.

What it actually builds

The output is a single model (OP_MODEL) that wraps three pieces, and it all happens inside this node on first run:

• An icon detector. A YOLO model trained to find clickable icons on a screen. The weights ship inside the repo as model.safetensors and get converted to a .pt file the first time you run the node. No download needed, just a few seconds of conversion.
• An icon captioner. A Florence-2 model finetuned to describe each detected icon ("camera icon", "submit button"). This one does not ship in the repo - the Loader pulls it from Hugging Face (microsoft/OmniParser, icon_caption_florence subfolder) on first load. That's a real download, so your first run is slow and needs network.
• A draw config picked from your platform choice, which just sets how thick the annotation boxes get drawn later.

The inputs, such as they are

This node is about as simple as a loader gets. Two inputs:

• repo - defaults to microsoft/OmniParser, which loads the official finetuned Florence-2 captioner. If you'd rather swap in any Florence-2 or BLIP-2 repo id, it'll accept that too; the node sniffs the string for "florence" or "blip". You'll almost never touch this.
• platform - pc, web, or mobile. It only changes annotation line thickness/scale in the drawn output, not the models. Pick what your screenshots actually are.

That's it. Wire model straight into OmniParser_Sampler's model input and forget it.

How to install

ComfyUI Manager is the easy route - search the pack title "ComfyUI_OmniParser" and let it run the requirements. Or do it by hand:

cd ComfyUI/custom_nodes
git clone https://github.com/smthemex/ComfyUI_OmniParser.git
pip install -r requirements.txt

Then restart ComfyUI. Note this is a heavy install for what's ultimately a vision pipeline: the requirements pull ultralytics, easyocr, supervision, and OpenCV on top of whatever torch/transformers you already have. That's the standard custom-node dependency pile-up - this is exactly the scenario the ComfyUI ecosystem docs mean when they talk about node conflicts breaking installs, so don't be surprised if Manager has a grumble or two.

Where people get burned

The README has an unusually specific warning: if ultralytics 8.3.41 got installed (Dec 2024), this pack breaks - check with pip show ultralytics and downgrade if you're on that build. Version drift like this is the price of a pack that pins its detection stack on another library's release cadence.

Also: if the Loader errors on a missing icon_caption_florence file or hangs, it's almost always the HF download failing partway. Re-run it; the conversion and download are idempotent. And if you're on CPU, expect this to be slow - the captioner is a real transformer, not a toy.

One more thing worth knowing: the pack is from smthemex, the person who turns "new model dropped on Hugging Face" into a ComfyUI node within days (TwinFlow, SenseNova U1, and plenty more). That means the code is current-but-fast-moving - the README's version pins matter, and you should update via Manager occasionally.