Trust building mechanism of multimodal interaction in autonomous driving system
DOI:
https://doi.org/10.71451/ISTAER2545Keywords:
Multimodal interaction; Autonomous driving; Trust building; Human-computer interaction; Explainability; User experienceAbstract
Multimodal interaction technology, as a key means of enhancing trust in autonomous driving systems, integrates multi-channel information such as vision, voice, and touch to build a transparent and explainable human-machine collaborative mechanism. This study systematically explored the role of multimodal interaction in building trust, analyzed interaction design strategies for different driving scenarios, and experimentally verified the significant effect of multimodal feedback on improving user trust. The results show that reasonable multimodal interaction design can enhance users' understanding and prediction of system behavior, reduce uncertainty anxiety, and accelerate trust restoration in abnormal situations. At the same time, the study also revealed issues such as sensor fusion accuracy and ethical privacy faced by current technologies and envisioned future development directions such as anthropomorphic interaction and contextual awareness. The results of this study provide a theoretical basis and practical guidance for the interaction design of autonomous driving systems, which is of great value in promoting the commercialization of the technology.
References
[1] Sun, X., & Zhang, Y. (2021). Improvement of autonomous vehicles trust through synesthetic-based multimodal interaction. IEEE Access, 9, 28213-28223. DOI: https://doi.org/10.1109/ACCESS.2021.3059071
[2] Qi, B., Guo, Q., & Liu, M. (2024, June). A study on the effects of different interaction modalities on driving trust in automated vehicles. In International Conference on Human-Computer Interaction (pp. 171-181). Cham: Springer Nature Switzerland. DOI: https://doi.org/10.1007/978-3-031-60477-5_13
[3] Ma, J., Zuo, Y., Du, H., Wang, Y., Tan, M., & Li, J. (2025). Interactive Output Modalities Design for Enhancement of User Trust Experience in Highly Autonomous Driving. International Journal of Human–Computer Interaction, 41(10), 6172-6190. DOI: https://doi.org/10.1080/10447318.2024.2375697
[4] Yi, B., Cao, H., Song, X., Wang, J., Guo, W., & Huang, Z. (2023). Measurement and real-time recognition of driver trust in conditionally automated vehicles: Using multimodal feature fusions network. Transportation research record, 2677(8), 311-330. DOI: https://doi.org/10.1177/03611981231156576
[5] Kaufman, R., Costa, J., & Kimani, E. (2024). Effects of multimodal explanations for autonomous driving on driving performance, cognitive load, expertise, confidence, and trust. Scientific reports, 14(1), 13061. DOI: https://doi.org/10.1038/s41598-024-62052-9
[6] Yang, Z., Song, N., Li, W., Zhu, X., Zhang, L., & Torr, P. H. (2025). Deepinteraction++: Multi-modality interaction for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence. DOI: https://doi.org/10.1109/TPAMI.2025.3565194
[7] Wang, W., Wang, L., Zhang, C., Liu, C., & Sun, L. (2022). Social interactions for autonomous driving: A review and perspectives. Foundations and Trends® in Robotics, 10(3-4), 198-376. DOI: https://doi.org/10.1561/2300000078
[8] Tekkesinoglu, S., Habibovic, A., & Kunze, L. (2025). Advancing explainable autonomous vehicle systems: A comprehensive review and research roadmap. ACM Transactions on Human-Robot Interaction, 14(3), 1-46. DOI: https://doi.org/10.1145/3714478
[9] Omeiza, D., Webb, H., Jirotka, M., & Kunze, L. (2021). Explanations in autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(8), 10142-10162. DOI: https://doi.org/10.1109/TITS.2021.3122865
[10] Capallera, M., Meteier, Q., De Salis, E., Widmer, M., Angelini, L., Carrino, S., ... & Mugellini, E. (2023). A contextual multimodal system for increasing situation awareness and takeover quality in conditionally automated driving. Ieee Access, 11, 5746-5771. DOI: https://doi.org/10.1109/ACCESS.2023.3236814
[11] Yang, J., Liang, N., Pitts, B. J., Prakah-Asante, K. O., Curry, R., Blommer, M., ... & Yu, D. (2023). Multimodal sensing and computational intelligence for situation awareness classification in autonomous driving. IEEE Transactions on Human-Machine Systems, 53(2), 270-281. DOI: https://doi.org/10.1109/THMS.2023.3234429
Downloads
Published
Issue
Section
License
Copyright (c) 2025 International Scientific Technical and Economic Research
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
This work is licensed under the Creative Commons Attribution International License (CC BY 4.0).
