· 5 min read

What Changes When the Output Moves

LLM, VLM and VLA look like a ladder of added inputs, but once a model's output is a motor command it needs control-loop speed, robot data and a different way to produce actions.

Glowing translucent geometric shapes, a cube, a sphere, two icosahedra and a cylinder, in blue, purple, green and orange, floating above a reflective dark surface.
  • Models
  • AI Agents

A vision model looks at a photo of a cluttered shelf and writes, correctly, that the blue box behind the bottle is the one you want. Give that sentence to a robot arm and nothing happens. The arm needs a position, a grip width and a moment to close its fingers, many times a second, and it needs them before the bottle tips over.

The usual way to explain the three acronyms is a ladder. A large language model (LLM) reads and writes text. A vision-language model (VLM) also takes images in. A vision-language-action model (VLA) also sends actions out. Each rung adds one thing.

If you are deciding whether any of this belongs in your operation, which layer you are buying matters more than which acronym is on the slide. The ladder is right about the inputs and misleading about almost everything else. Once the output is a motor command, the model sits inside a control loop: it has to act many times a second, it learns its skills from robot data nobody has scraped off the web, and it cannot write those actions the way it writes words. Some of the leading systems now split the job back apart, with a VLM doing the thinking and a VLA doing the moving.

The first VLAs really were language models

The term comes from Google DeepMind’s RT-2 paper in July 2023. Its trick was to treat robot movement as one more language: “we express the actions as text tokens and incorporate them directly into the training set of the model in the same way as natural language tokens”. Each movement dimension was cut into 256 bins, so an arm command became a short string of numbers the model could write the same way it writes words.

It worked better than it had any right to. Web pretraining let the robot handle new objects and follow commands that never appeared in its robot data. The paper was also clear about where that stopped. In its limitations section the authors write that “the robot does not acquire any ability to perform new motions by virtue of including this additional experience”. Web data could tell the model which object to pick up. How to move its arm was still learned from robot data.

Speed was the other problem. The team ran RT-2 in a cloud service that the robot queried over the network. Its largest version, at 55 billion parameters, managed 1 to 3 commands a second. That is fine for picking up a can and hopeless for folding a shirt.

Actions turned out not to be words

Physical Intelligence’s π0, published in October 2024, kept a pretrained VLM and stopped asking it to write actions as tokens. It added a separate action expert, 300 million parameters next to a 3-billion-parameter open VLM, that produces continuous movements directly, using a technique related to the diffusion methods behind image generators. Because it predicts a chunk of movements at once, it can drive a robot “at frequencies of up to 50 Hz for dexterous tasks such as laundry folding”.

The same team later spelled out why writing actions as tokens breaks down. Their knowledge insulation paper says “autoregressive decoding of discrete tokens is poorly suited to this kind of high-frequency continuous control”, because the bins are coarse and generating one token at a time with a large model is slow.

They also found that bolting on the action expert damaged the VLM underneath. The paper reports that the π0 recipe “leads to degradation in terms of both language following and training speed, as the gradients from the action expert degrade the pre-trained VLM backbone”. Teach a model to move, carelessly, and it gets worse at understanding what you asked it to do. Their fix is to shield the VLM’s weights from the action expert during training. So a VLA keeps a VLM’s understanding only if its builders protect it.

Data is the deepest difference. π0 was pre-trained on “over 10,000 hours of robot data”, collected on real machines doing real tasks. Text and images exist by the trillion online. Demonstrations of a particular gripper loading a particular fixture exist only if someone records them, and RT-2’s limitation still holds. The robot still needs robot data to learn the physical skills themselves.

The VLM comes back as the planner

Google DeepMind’s current design makes the split explicit. Gemini Robotics 2, announced in July, ships as separate models. Gemini Robotics ER 2 “is a vision language model (VLM) that acts as our agent”. It “reasons about the steps needed to complete the task, coordinates with the VLA to carry out the actions, and tracks progress until the task is done”. The VLA converts that plan into motor control. Google’s own write-up concedes that “multi-finger dexterous manipulation remains challenging”. The two halves also reach customers differently. Any developer can try the planning model in Google AI Studio, while the VLA models are limited to early-access partners, and so is Gemini Robotics 2.5, announced on 10 November.

Physical Intelligence, the company behind π0, has taken a different route to customers. It published the weights for π0 and π0.5 in its openpi repository, and none for the models that followed. Its newest models now sit behind a hosted API, in beta since 13 October. Developers upload demonstrations, fine-tune on the company’s compute and call the result over the network. The company’s stated reason for hosting is that it “can update the models underpinning the API rapidly”, and it adds that “the more our models are used, the more we learn how to improve them”. That brings a VLA closer to a service you call. You still bring the robot and the demonstrations, and the same post reports “network connectivity challenges for some” of the early deployments, the cloud dependence RT-2 had in 2023.

An LLM or a VLM is a service you call, testable on your own documents and images by Friday. Even as a hosted service, a VLA needs your hardware and a stream of demonstrations from your own floor, and the only meaningful test is a robot doing the task in your space. The useful test is whether the model has to close the control loop itself. If it only needs to inspect a scene, interpret it or produce a plan that a person or your existing automation carries out, you may not need a VLA at all. That half is useful and you can buy it now.

So before anyone pitches you a VLA, ask where in your operation something physically has to move, and who is going to record the hours of demonstrations that teach a model to move it.