Before you set down a heavy, fragile pitcher of water, you don't just send the motor commands and hope. You run a quick, mostly unconscious simulation first - if the grip is too loose, it slips; if you knock the rim, it spills - and you adjust before contact happens, not after. Most robot policies don't get to do this. They map what they see directly to what they do, with nothing in between that lets them check a plan against its consequences before committing to it.
A cluster of papers from the last few months are building that missing step: a model that watches the scene, imagines a few seconds forward, and only then decides what to do.
From predicting pixels to predicting consequences
The oldest version of this idea is a video model: given the current frame and a planned action, predict what the next frame looks like. That alone doesn't make a policy - you need something that turns “here's what will probably happen” into “here's what to do about it.” The recent work differs mainly in how tightly it fuses those two things.
τ0-WM, from AgiBot, builds a single video-action model that does both jobs off one shared representation: a policy interface that predicts executable action chunks, and an action-conditioned video simulator that imagines future observations and their consequences. At test time it runs a propose–evaluate–revise loop - generate a candidate action, imagine its outcome, and refine before executing - spending extra computation exactly where a bad prediction would be costly.
Cosmos Policy, from NVIDIA and Stanford, takes a different route to the same place: instead of building a new world model from scratch, they fine-tune Cosmos-Predict2, an existing large pretrained video generator, directly into a robot policy - no new architecture, a single stage of post-training on the target robot's own demonstration data. Robot actions are encoded as latent frames inside the video model's own diffusion process, so the model that already understands physical plausibility and temporal coherence from internet-scale video becomes the same model that outputs actions. It reports state-of-the-art results on the LIBERO and RoboCasa simulation benchmarks and the strongest scores among the methods tested on real-world bimanual tasks - and, notably, it can keep improving after deployment by feeding its own rollout data back in to refine both the world model and a value function for model-based planning.
GigaBrain-0 and WorldVLA sit in the same family - vision-language-action policies where a world model isn't a side component but part of the backbone the action decisions are conditioned on, rather than a separate module bolted on after the fact.
Why this is different from “just add a simulator”
It would be easy to mistake this for restating what a physics simulator already does - predict forward, check the result. The distinction is where the physics comes from. A hand-built simulator needs someone to specify contact models, friction coefficients, and material properties in advance, and it's only ever as accurate as those specifications. A video world model learns its physical priors from watching millions of real videos, so it picks up things like deformation, liquid behavior, and cluttered-scene occlusion that nobody explicitly programmed in - at the cost of the predictions being a learned approximation rather than a guaranteed simulation. The newer papers are explicit that this is still an open tradeoff, not a solved one: predictions degrade over longer horizons, and “imagined” isn't the same guarantee as “computed.”
Where this connects for us
This is a close cousin to what Sim Fabric already does, just running in a different place. Sim Fabric composes a world offline, before a task ever starts, so a simulator has something physically consistent to load. A video world model does the analogous thing online, mid-task, imagining one step ahead instead of composing a whole scene ahead of time. The offline and online versions of “imagine before you commit” are, we think, going to end up as two ends of the same problem - one giving you a world to start from, the other giving you a way to check your next move against it before you make it.