VLA Observation Builder

A VLA policy doesn't take "an image" - it takes an observation: the task, the current joint state, and one or more camera views, all packed into the exact structure the model expects. VLA Observation Builder is the node that assembles and validates that package. Think of it as the polite interface between your ComfyUI graph and a policy server that demands a very specific dictionary.

The required inputs are the ones that define the moment: task (the instruction, like "Pick up the object and place it in the container"), state_json (the current robot state as a JSON array, default [0, 0, 0, 0, 0, 0, 0, 0]), and primary_image - the main camera frame, with primary_camera naming the key it maps to (default observation.images.front, which is the DROID/LeRobot convention). history_fps and timestamp annotate the observation so temporal information is honest. The optional sockets add the classic multi-camera setup: wrist_image, secondary_image, and goal_image (the target state you want the robot to reach), each with its own camera key.

Here's the genuinely clever bit: an IMAGE batch on any camera socket becomes temporal history. Connect a stack of frames and the node treats them as a time series at history_fps, which is exactly what recurrent/temporal policies consume - instead of feeding one frame, you feed the last N frames and let the model see motion. The history_frames output tells you how many frames ended up in the observation.

It also validates as it builds. State dimensions are checked against the embodiment profile if you connect one, so a 7-value state against an 8-DoF profile fails loudly instead of silently confusing a policy server. Outputs are observation (the typed VLA_OBSERVATION you wire into a policy node), observation_summary (a human-readable string), and history_frames.

Installing it

Nothing beyond the pack - this node loads no policy and no robot SDK, by design. Install via ComfyUI Manager (search "ComfyUI VLM nodes") or:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
python -m pip install -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements.txt

No extra robotics dependencies needed unless you're running the openpi or GR00T policy clients downstream, which use requirements-robotics-client.txt.

Common issues

• Policy rejects the observation → nine times out of ten it's a key or dimension mismatch: your primary_camera string doesn't match what the policy expects (observation.images.front is the DROID convention, but check yours). Connect an embodiment profile so the builder can catch dimension drift.
• Wrong number of state values → state_json length must match the embodiment's state_dim. Count your brackets.
• Video history ignored → the batch must actually be multiple frames; a single-frame IMAGE batch is just a still.
• "It runs but the robot does nothing" → the observation is valid but the policy didn't see what you think it saw. Check observation_summary - it's there precisely so you can see what got packed before it ships.

The workflow shape to keep in mind: Embodiment Profile → Observation Builder → Policy node → Safety Gate. This node is step two, and it's the one that makes sure step three gets a well-formed message.