Evolutionary Decision-Making and Planning for Autonomous Driving Based on Safe and Rational Exploration and Exploitation

Kang Yuan, Yanjun Huang, Shuo Yang, Zewei Zhou, Yulei Wang, Dongpu Cao, Hong Chen

Engineering ›› 2024, Vol. 33 ›› Issue (2) : 108-120.

PDF(5398 KB)
PDF(5398 KB)
Engineering ›› 2024, Vol. 33 ›› Issue (2) : 108-120. DOI: 10.1016/j.eng.2023.03.018
Research
Article

Evolutionary Decision-Making and Planning for Autonomous Driving Based on Safe and Rational Exploration and Exploitation

Author information +
History +

Abstract

Decision-making and motion planning are extremely important in autonomous driving to ensure safe driving in a real-world environment. This study proposes an online evolutionary decision-making and motion planning framework for autonomous driving based on a hybrid data- and model-driven method. First, a data-driven decision-making module based on deep reinforcement learning (DRL) is developed to pursue a rational driving performance as much as possible. Then, model predictive control (MPC) is employed to execute both longitudinal and lateral motion planning tasks. Multiple constraints are defined according to the vehicle’s physical limit to meet the driving task requirements. Finally, two principles of safety and rationality for the self-evolution of autonomous driving are proposed. A motion envelope is established and embedded into a rational exploration and exploitation scheme, which filters out unreasonable experiences by masking unsafe actions so as to collect high-quality training data for the DRL agent. Experiments with a high-fidelity vehicle model and MATLAB/Simulink co-simulation environment are conducted, and the results show that the proposed online-evolution framework is able to generate safer, more rational, and more efficient driving action in a real-world environment.

Graphical abstract

Keywords

Autonomous driving / Decision-making / Motion planning / Deep reinforcement learning / Model predictive control

Cite this article

EndNote

Ris (Procite)

Bibtex

Download citation ▾
Kang Yuan, Yanjun Huang, Shuo Yang, Zewei Zhou, Yulei Wang, Dongpu Cao, Hong Chen. Evolutionary Decision-Making and Planning for Autonomous Driving Based on Safe and Rational Exploration and Exploitation. Engineering, 2024, 33(2): 108‒120 https://doi.org/10.1016/j.eng.2023.03.018

References

Publishing order | Descend order by publishing year | Descend order by cited within
[1]
J. Nilsson, M. Brännström, E. Coelingh, J. Fredriksson. Lane change maneuvers for automated vehicles. IEEE Trans Intell Transp Syst, 18 (5) (2016), pp. 1087-1096
[2]
X. Wang, X. Qi, P. Wang, J. Yang. Decision making framework for autonomous vehicles driving behavior in complex scenarios via hierarchical state machine. Auton Intell Syst, 1 (1) (2021), pp. 1-12
[3]
S. Noh. Decision-making framework for autonomous driving at road intersections: safeguarding against collision, overly conservative behavior, and violation vehicles. IEEE Trans Ind Electron, 66 (4) (2018), pp. 3275-3286
[4]
J. Nilsson, J. Sjöberg. Strategic decision making for automated driving on two-lane, one way roads using model predictive control. Proceedings of IEEE Intelligent Vehicles Symposium (IV); 2013 Jun 23-26, IEEE, Gold Coast, QLD, Australia. New York City (2013), pp. 1253-1258
[5]
Y. Du, Y. Wang, C.Y. Chan. Autonomous lane-change controller via mixed logical dynamical. Proceedings of 17th International IEEE Conference on Intelligent Transportation Systems (ITSC); 2014 Oct 8-11, IEEE, Qingdao, China. New York City (2014), pp. 1154-1159
[6]
Z. Zhou, Z. Yang, Y. Zhang, Y. Huang, H. Chen, Z. Yu. A comprehensive study of speed prediction in transportation system: from vehicle to traffic. iScience, 25 (3) (2022), Article 103909
[7]
J. Karlsson, N. Murgovski, J. Sjöberg. Optimal trajectory planning and decision making in lane change maneuvers near a highway exit. Proceedings of 18th European Control Conference (ECC); 2019 Jun 25-28, IEEE, Naples, Italy. New York City (2019), pp. 3254-3260
[8]
J. Nilsson, J. Silvlin, M. Brannstrom, E. Coelingh, J. Fredriksson. If, when, and how to perform lane change maneuvers on highways. IEEE Intell Transp Syst Magazine, 8 (4) (2016), pp. 68-78
[9]
Z. Cui, J. Hu, H. Guan. A lane-changing trajectory planning and assistant decision-making method for autonomous vehicle. Proceedings of 18th COTA International Conference of Transportation Professionals (CICTP); 2018 Jul 05-08, ASCE, Beijing, China. Reston (2018), pp. 87-101
[10]
D. Xu, Z. Ding, X. He, H. Zhao, M. Moze, F. Aioun, et al.. Learning from naturalistic driving data for human-like autonomous highway driving. IEEE Trans Intell Transp Syst, 22 (12) (2020), pp. 7341-7354
[11]
Y. Liu, B. Zhou, X. Wang, L. Li, S. Cheng, Z. Chen, et al.. Dynamic lane-changing trajectory planning for autonomous vehicles based on discrete global trajectory. IEEE Trans Intell Transp Syst, 23 (7) (2022), pp. 8513-8527
[12]
R. Van Hoek, J. Ploeg, H. Nijmeijer. Cooperative driving of automated vehicles using B-splines for trajectory planning. IEEE Trans Intell Vehicles, 6 (3) (2021), pp. 594-604
[13]
D. Kim, Y. Jeong, C.C. Chung. Lateral vehicle trajectory planning using a model predictive control scheme for an automated perpendicular parking system. IEEE Trans Ind Electron, 70 (2) (2023), pp. 1820-1829
[14]
T.A. Mai, T.S. Dang, D.T. Duong, V.C. Le, S. Banerjee. A combined backstepping and adaptive fuzzy PID approach for trajectory tracking of autonomous mobile robots. J Braz Soc Mech Sci Eng, 43 (3) (2021), pp. 1-13
[15]
A.J. Moshayedi, J. Li, L. Liao. Simulation study and PID tune of automated guided vehicles (AGV). Proceedings of IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA); 2021 Jun 18-20, IEEE, Hong Kong, China. New York City (2021), pp. 1-7
[16]
A.D. Sabiha, M.A. Kamel, E. Said, W.M. Hussein. ROS-based trajectory tracking control for autonomous tracked vehicle using optimized backstepping and sliding mode control. Robot Auton Syst, 152 (2022), Article 104058
[17]
H. El Atwi, N. Daher. A composite model predictive and super twisting sliding mode controller for stable and robust trajectory tracking of autonomous ground vehicles. Proceedings of IEEE 3rd International Multidisciplinary Conference on Engineering Technology (IMCET); 2021 Dec 8-10, IEEE, Beirut, Lebanon. New York City (2022), pp. 107-112
[18]
J. Ji, A. Khajepour, W.W. Melek, Y. Huang. Path planning and tracking for vehicle collision avoidance based on model predictive control with multiconstraints. IEEE Trans Vehicular Technol, 66 (2) (2016), pp. 952-964
[19]
Y. Huang, H. Wang, A. Khajepour, H. Ding, K. Yuan, Y. Qin. A novel local motion planning framework for autonomous vehicles based on resistance network and model predictive control. IEEE Trans Vehicular Technol, 69 (1) (2019), pp. 55-66
[20]
A. Wischnewski, T. Herrmann, F. Werner, B. Lohmann. A tube-MPC approach to autonomous multi-vehicle racing on high-speed ovals. IEEE Trans Intell Vehicles, 8 (1) (2023), pp. 368-378
[21]
Evens B, Schuurmans M, Patrinos P. Learning MPC for interaction-aware autonomous driving: a game-theoretic approach. 2021. arXiv:2111.08331.
[22]
K. Yuan, H. Shu, Y. Huang, Y. Zhang, A. Khajepour, L. Zhang. Mixed local motion planning and tracking control framework for autonomous vehicles based on model predictive control. IET Intell Transp Syst, 13 (6) (2019), pp. 950-959
[23]
F. Mohseni, E. Frisk, L. Nielsen. Distributed cooperative MPC for autonomous driving in different traffic scenarios. IEEE Trans Intell Vehicles, 6 (2) (2020), pp. 299-309
[24]
Y. Huang, H. Ding, Y. Zhang, H. Wang, D. Cao, N. Xu, et al.. A motion planning and tracking framework for autonomous vehicles based on artificial potential field elaborated resistance network approach. IEEE Trans Ind Electron, 67 (2) (2020), pp. 1376-1386
[25]
Q. Zhou, D. Zhao, B. Shuai, Y. Li, H. Williams, H. Xu. Knowledge implementation and transfer with an adaptive learning network for real-time power management of the plug-in hybrid vehicle. IEEE Trans Neural Netw Learn Syst, 32 (12) (2021), pp. 5298-5308
[26]
Y. Wang, D. Zhang, J. Wang, Z. Chen, Y. Li, Y. Wang, et al.. Imitation learning of hierarchical driving model: from continuous intention to continuous trajectory. IEEE Robot Autom Lett, 6 (2) (2021), pp. 2477-2484
[27]
C.J. Hoel, K. Wolff, L. Laine. Automated speed and lane change decision making using deep reinforcement learning. Proceedings of 21st International Conference on Intelligent Transportation Systems (ITSC); 2018 Nov 4-7, IEEE, Maui, HI, USA. New York City (2018), pp. 2148-2155
[28]
Y. Liu, X. Wang, L. Li, S. Cheng, Z. Chen. A novel lane change decision-making model of autonomous vehicle based on support vector machine. IEEE Access, 7 (2019), pp. 26543-26550
[29]
X. Wang, J. Wu, Y. Gu, H. Sun, L. Xu, S. Kamijo, et al.. Human-like maneuver decision using LSTM-CRF model for on-road self-driving. Proceedings of 21st International Conference on Intelligent Transportation Systems (ITSC); 2018 Nov 4-7, IEEE, Maui, HI, USA. New York (2018), pp. 210-216
[30]
Y. Xiao, F. Codevilla, A. Gurram, O. Urfalioglu, A.M. L’opez. Multimodal end-to-end autonomous driving. IEEE Trans Intell Transp Syst, 23 (1) (2020), pp. 537-547
[31]
M. Menner, K. Berntorp, M.N. Zeilinger, S. Di Cairano. Inverse learning for data-driven calibration of model-based statistical path planning. IEEE Trans Intell Vehicles, 6 (1) (2020), pp. 131-145
[32]
B. Peng, Q. Sun, S.E. Li, D. Kum, Y. Yin, J. Wei, et al.. End-to-end autonomous driving through dueling double deep Q-network. Automotive Innovation, 4 (3) (2021), pp. 328-337
[33]
Y. Lin, J. McPhee, N.L. Azad. Comparison of deep reinforcement learning and model predictive control for adaptive cruise control. IEEE Trans Intell Vehicles, 6 (2) (2020), pp. 221-231
[34]
X. He, H. Yang, Z. Hu, C. Lv. Robust lane change decision making for autonomous vehicles: an observation adversarial reinforcement learning approach. IEEE Trans Intell Vehicles, 8 (1) (2023), pp. 184-193
[35]
G. Li, S. Li, S. Li, Y. Qin, D. Cao, X. Qu, et al.. Deep reinforcement learning enabled decision-making for autonomous driving at intersections. Automotive Innovation, 3 (4) (2020), pp. 374-385
[36]
Z. Liu, J. Hu, T. Song, Z. Huang. A methodology based on deep reinforcement learning to autonomous driving with double Q-Learning. Proceedings of 7th International Conference on Computer and Communications (ICCC); 2021 Dec 10-13, IEEE, Chengdu, China. New York City (2022), pp. 1266-1271
[37]
S. Aradi. Survey of deep reinforcement learning for motion planning of autonomous vehicles. IEEE Trans Intell Transp Syst, 23 (2) (2022), pp. 740-759
[38]
G. Li, Y. Yang, S. Li, X. Qu, N. Lyu, S.E. Li. Decision making of autonomous vehicles in lane change scenarios: deep reinforcement learning approaches with risk awareness. Transp Res Part C Emerg Technol, 134 (2022), Article 103452
[39]
Y. Zhang, P. Sun, Y. Yin, L. Lin, X. Wang. Human-like autonomous vehicle speed control by deep reinforcement learning with double Q-learning. Proceedings of IEEE Intelligent Vehicles Symposium (IV); 2018 Jun 26-30; Changshu, China, IEEE, Changshu, China. New York City (2018), pp. 1251-1256
AI Summary AI Mindmap
PDF(5398 KB)

Accesses

Citations

Detail
Sections
Recommended

/

〈 〉