Large language models have transformed interactions with computers, but their ability to understand and operate in the physical world remains limited. Researchers believe world models could overcome this barrier by enabling AI systems to predict how environments will change in response to specific actions, tells Live Science (full article available to subscribers).
A world model creates a mathematical representation of an environment, such as a warehouse, video game, or physical workspace. Unlike large language models, which primarily predict the next word in a sequence, world models perform action-conditioned future prediction. They first estimate the current state of an environment and then calculate how an action could change that state. This ability could support applications in robotics, autonomous driving, and scientific discovery.
The concept has roots in cognitive science and control theory, but modern world models use neural networks trained on large datasets. Training commonly relies on video paired with action information, although lidar, 3D data, audio, and text can also contribute. By studying sequences of environmental states and actions, AI learns statistical relationships between an action and its consequences.
Researchers are exploring several ways to represent this information. Some models work directly with pixels, essentially predicting future visual scenes. Others construct 3D representations that capture spatial relationships. Another approach performs predictions within a model’s latent space, where objects and environments are represented mathematically rather than reconstructed pixel by pixel. This method can reduce unnecessary computation, although visual predictions are generally easier for humans to inspect and troubleshoot.
Major challenges remain. High-quality video paired with action data is difficult to collect, and researchers have not agreed on the best neural network architecture. Current models also lack the continuous learning abilities humans and animals develop through real-world interaction.
If researchers overcome these limitations, world models could help AI move beyond generating information toward understanding environments, anticipating consequences, and making informed decisions within the physical world.