Researchers at Delft University of Technology have developed a system that could let passengers adjust an autonomous vehicle’s driving style simply by talking to it. The approach uses a large language model to translate natural-language requests into changes to the vehicle’s motion-planning system, tells IEEE Spectrum.
Autonomous vehicles rely on motion planners to select safe and efficient paths through traffic. Engineers typically configure parameters such as speed, acceleration, steering, and ride smoothness before deployment. This leaves passengers with little ability to modify vehicle behavior based on changing preferences, such as wanting a gentler ride when feeling carsick or faster progress when running late.
The TU Delft system uses OpenAI’s GPT-4o-mini to interpret such requests. Importantly, the LLM does not directly control the vehicle. Instead, it evaluates the relative importance of criteria used by a model predictive path integral controller, including speed, steering angle, and collision probability. These preferences adjust parameters around safety limits established by researchers.
Keeping the LLM separate from the primary controller addresses limitations associated with using language models for real-time driving decisions. LLMs can respond too slowly for rapidly changing traffic situations and cannot provide the performance guarantees available from deterministic motion planners.
The system also keeps passengers involved. Before changing driving behavior, it explains its intended adjustments in nontechnical language and requests confirmation. Passengers can reject the proposal or provide additional instructions if the system misinterprets their preferences.
Researchers tested the approach using the nuPlan autonomous-driving simulator in highway-merging scenarios. Across eight prompts, the system adjusted motion-planning parameters according to user intent. Requests for comfort produced smoother driving, while expressions of urgency resulted in higher speeds.
The research suggests LLMs may be most useful in autonomous vehicles as an interface between people and established control systems rather than as direct drivers. However, experts note that additional mathematical safeguards may be necessary to verify safety before LLM-influenced decisions are implemented.
