VLA Policy — Universal HTTP

If you only wire up one robotics policy node, make it this one. VLA Policy - Universal HTTP is the pack's workhorse: it takes a VLA_OBSERVATION, POSTs it to a policy server, and returns the action chunk. It's "universal" because it talks to the pack's own sidecar server, which loads whichever policy you've chosen (SmolVLA, π0, whatever) in an isolated LeRobot environment and exposes it over a simple HTTP endpoint. One node, one protocol, any policy behind it.

The flow works like this: you run the policy outside ComfyUI in its own environment - the pack's lerobot_policy_server.py script is the reference implementation. That sidecar loads the chosen policy and its serialized processors, keeps the model resident for low latency, offloads to CPU after an idle timeout, and answers one request at a time (preventing stateful policy races). It uses bounded JSON/JPEG rather than pickle, and it never returns tracebacks, env vars, or auth headers. This node is the clean client for that server.

Inputs: observation (from the Observation Builder), endpoint (default http://127.0.0.1:8787 - the sidecar's default port), timeout_seconds (120), include_history (default true - whether the full temporal observation gets sent), and allow_remote (default false). That last one is a security gate: pointing at a remote server requires HTTPS and explicit opt-in. The auth token (VLA_POLICY_TOKEN) is never a workflow input - you set the same value in the environment that launches ComfyUI and the environment that runs the sidecar, and the node resolves it server-side. If the two don't match, you get a clean auth failure instead of a confusing hang.

Outputs: actions (the typed VLA_ACTIONS chunk - wire it into the safety gate, inspect node, or a simulator) and inference_report (a JSON string with what the server did, including timing and the action chunk summary).

Installing it

The HTTP client needs nothing beyond the main pack:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
python -m pip install -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements.txt

(ComfyUI Manager, search "ComfyUI VLM nodes", works too.) The server side is a separate environment - see examples/robotics/README.md for the WSL/D-drive setup, and tests/manual_robotics_smoke.py for a real end-to-end test against a running sidecar.

Common issues

• Connection refused → the sidecar isn't running, or it's on a different machine/port. Confirm the endpoint, and remember remote URLs need HTTPS + allow_remote=true.
• Auth failures out of nowhere → VLA_POLICY_TOKEN mismatch between the ComfyUI environment and the sidecar environment. Set the same token in both.
• "No such policy type" → the sidecar, not the node, knows what policies it can load. Check the policy server's logs; the node is just the messenger.
• Slow responses → the first call after idle can be slow because the model offloads to CPU. That's the idle-offload feature working; bump the idle timeout in the sidecar if it annoys you.

Start with SmolVLA on consumer hardware, per the pack's own guidance - it's the fast path, and this node makes swapping the backend a config change rather than a workflow rewrite.