Robots learn best through exposure, seeing how objects sit, move, or collide in varied settings. But gathering real-world data is costly, time consuming, and limited in variety. To overcome this, MIT’s CSAIL (in collaboration with Toyota Research Institute) developed a generative AI system called steerable scene generation to automatically build rich training grounds for robotic agents, tells MIT News.
The system begins with a library of over 44 million 3D room scenes populated with everyday items, such as tables, plates, books, and cups. It uses diffusion models to generate new scenes, and then guides (or “steers”) the output toward physically plausible, useful configurations. A core steering technique is Monte Carlo Tree Search (MCTS), where the AI explores multiple scene variants and picks arrangements that best satisfy objectives such as realism, object count, or layout diversity.
One advantage of this approach is that it can push beyond the biases of the original training set. For instance, the system managed to place 34 items in a restaurant scene, far above the average 17 in its base data, by exploring creative layouts. It also supports conditional prompting: users can ask for “a messy breakfast table” or “a kitchen with a bowl of apples,” and the system can generate them with high accuracy (98% in some tests) while avoiding common visual glitches such as clipping.
Once scenes are generated, virtual robots can practice manipulation tasks, such as arranging cutlery, stacking dishes, and repositioning objects, in environments that more closely mimic the real world. Because the scenes are diverse, robots trained this way may generalize better when moved into physical settings.
The authors view this work as a proof of concept. Their next steps include expanding the object library, creating new assets (not just rearranging existing ones), and modeling articulated parts (drawers, jars, cabinets) that open or twist. If successful, this approach could narrow the notorious “simulation-to-reality gap” and accelerate the pace of robot learning in homes, factories, and beyond.