Checkbin Start Run

What this node actually is

Checkbin Start Run is the front door to Checkbin, a human-preference A/B testing service for AI outputs. It's the same platform behind those big r/StableDiffusion "showdown" posts - Flux-ControlNet-Upscaler versus the field, SUPIR versus Flux, LTX versus HunyuanVideo - where the author submits pairs of images to a shareable link and real people vote on which looks better. This pack is that voting platform bolted onto ComfyUI, and Start Run is where you begin.

If you've spent any time comparing two upscalers or two checkpoints by eye, you know the problem it solves: your gut says one is sharper, your monitor says the other, and your tired retinas are not a statistical test. Checkbin gets you actual preference data with a link you can hand to anyone. That's the pitch.

How it works

Start Run is the one node in the pack that talks to Checkbin's servers directly. You hand it a token, it authenticates, opens a "run" against an app you've defined, and hands you one or more CHECKBIN bins. Those bins then flow through the Save Image / Save String nodes (where you attach your candidates) and finally into Submit Bin, which uploads the whole thing for human scoring.

The mechanism is right there in run_node.py: it calls checkbin.authenticate(token=token), builds an App from app_key, and then runs a context manager checkbin_app.start_run(...) that yields a list of bins. One nuance that surprises people: this node always re-executes. Its IS_CHANGED returns NaN, so ComfyUI treats it as permanently changed and re-runs it every single queue - which means every generate hits the API. That's expected, but don't put it in a loop you plan to spam.

The inputs that matter

Only one input is required: token. It's a string you paste from app.checkbin.dev after signing up - the example workflow even embeds the sign-up link in a Note node. Without a valid token, nothing else works.

The optional inputs are where the power is:

• directory - point this at a local folder of images and Start Run auto-uploads every .jpg, .jpeg, .png, or .webp in it as an input set. That's how the shipped example feeds a folder of test photos into the comparison. It errors with a FileNotFoundError if the folder doesn't exist or turns up empty.
• app_key - identifies which "app" (test project) this run belongs to; defaults to comfyui. The example sets it to compare_upscalers.
• checkin_id / set_id / run_id - for targeting specific existing inputs or resuming a specific run instead of auto-creating one.
• sample_size - if you have a big input set and only want to test a random subset, cap it here. Setting it to 0 means "no limit".

The output

Start Run returns one CHECKBIN bin - but it's a list (the node declares OUTPUT_IS_LIST). Give it a directory with ten images and you get ten bins, one per input. ComfyUI fans the list out to the nodes wired below it, so your Save/Submit chain executes once per bin. That's the trick that lets one workflow generate a whole comparison set in a single run.

Installing it

It ships in the checkbin-comfy pack alongside the other five nodes, so one install covers everything:

cd ComfyUI/custom_nodes
git clone https://github.com/checkbins/checkbin-comfy

Restart ComfyUI after cloning. Or skip the terminal: open ComfyUI Manager → Install Custom Nodes and search for "checkbin-comfy". The only Python dependency is the checkbin package (it's the sole line in requirements.txt), which Manager installs for you. There are no model downloads - this is a thin HTTP client, not a diffusion model, so it costs you nothing in disk or VRAM beyond the API calls.

Common issues

• Auth errors on every run - a missing, mistyped, or expired token fails at this node before anything else can. Grab a fresh one from the Checkbin dashboard.
• Directory 'X' cannot be found. - Start Run validates the folder before it does anything; if it's empty or the path is wrong, that's your error. It also silently ignores non-image files.
• It runs even when nothing changed - that's the NaN IS_CHANGED doing its job. If your queue suddenly stalls on API calls, this node is why. It's by design, but worth knowing when you're testing a workflow and re-queueing constantly.