OmniParser_Sampler

This is the node that actually does OmniParser's thing. Give it a screenshot plus the model bundle from OmniParser_Loader, and it returns two things: an annotated copy of the image with every text region and icon boxed and numbered, and a plain-text list that says what each box is. That text output is the whole point - it's the part you hand to an LLM so it can "look" at a GUI and decide what to click. Think of it as set-of-marks prompting, except the marks are computed locally instead of being drawn by a proprietary vision model.

How it works

Three stages, run in one pass:

1. OCR finds every text region and reads it ("Settings", "Sign in", whatever's on screen).
2. YOLO detects the icons - buttons, logos, chrome that has no text.
3. Florence-2 (the finetuned icon captioner the Loader fetched) describes each icon crop in a phrase, so an icon becomes "back arrow icon" instead of a mystery box.

Then overlapping detections get merged, numbered boxes get drawn on the image, and the outputs are produced: image (the annotated screenshot) and string (lines like IMG0's Text Box ID 1: Settings, IMG0's Icon Box ID 12: back arrow icon). Pipe that string into any text/LLM node and you have a GUI agent front end. Or just use it as a free accessibility-style screenshot annotator.

The inputs that matter

Nine inputs, but only a few you'll actually touch:

• box_threshold - icon detection confidence. Default 0.05 is very low on purpose: OmniParser wants to catch everything. If your output is cluttered with junk boxes, raise it to ~0.1–0.3.
• text_threshold - OCR confidence, default 0.9. High by design; drop it if real labels are being skipped.
• iou_threshold - how aggressively overlapping boxes get merged. Default 0.1 is conservative; raise it if you see double-boxes.
• orc_model / orc_mode - yes, that's "OCR" misspelled in the field names. easyocr is the default and is what requirements.txt actually installs. paddleocr is a second path you have to set up yourself - see below. orc_mode is just the language.
• output_bb_format (xyxy vs xywh) and output_coord_in_ratio - affect how coordinates appear in the string output; leave xyxy and true unless your downstream LLM wants pixels.
• paragraph groups OCR text into paragraphs; display_img is a debug toggle that pops up a window showing the raw OCR boxes.

The paddleocr trap

The README's requirements file has paddleocr and paddlepaddle commented out. So if you switch orc_model to paddleocr, it will just crash until you do this yourself:

pip install paddlepaddle paddleocr

Even then, note the code deliberately runs PaddleOCR on CPU because its CUDA path conflicts with PyTorch in the same process. For most people easyocr is the sane choice anyway. And heads up: easyocr downloads its own recognition weights the first time it runs, so the very first Sampler pass is slow.

Installation and gotchas

Same install as the Loader: ComfyUI Manager (search "ComfyUI_OmniParser"), or git clone https://github.com/smthemex/ComfyUI_OmniParser.git into ComfyUI/custom_nodes, pip install -r requirements.txt, restart. The pack's real dependency load is ultralytics + easyocr + supervision + OpenCV - heavyweight for a node that's arguably not image generation at all, and a textbook case of the custom-node dependency pile-up that the ecosystem has been complaining about for years.

The one version landmine the README calls out: if ultralytics is exactly 8.3.41 (a Dec 2024 release), this pack breaks - check pip show ultralytics and pin it to something else. And if your Sampler output is nonsense text, double-check you're feeding it the model from the Loader, not a random image - the two nodes are designed as a pair and don't work apart.