06/26/2023 -> 010/11/2023
Tony Wang's summer research internship for LLMs + Autonomous driving
- Advisor: Prof. Wei Xiang, College of Computer Science and Technology, ZJU
- Collabrating with Yingying Huang, Ziyue Lei and Qi Zheng
- Tech lead of the Visionary Co-Driver system, an LLMs agent enhancing the driver's risk perception on the roadside.
Automated driving invokes challenges in perceiving risks from road users, involving both risks from vehicles on road and potential risks on the roadside, such as pedestrian's jaywalking, and bicycle rushing from the corner. These risks have a degree of uncertainty and require caution in advance. Existing Driver Assistance Systems (DASs) focus on risks on road, and cannot handle potential risks on the roadside appropriately. This paper introduces Visionary Co-Driver, a system that intergrates Large Language Model (LLM) and Augmented Reality Heads-Up Display (AR-HUD) to enhance drivers' risk perception. The system combines video processing algorithms and LLMs to analyze road scene information and identify potential risky pedestrians on the roadside. These risks are dynamically projected to an adaptive AR-HUD interface utilizing eye-tracking technology to enhance drivers' attention. A user study validates that Visionary Co-Driver improves risk perception over conventional AR-HUD, enhancing automated driving safety.