[1]	ISO 21448: Road vehicles—safety of the intended functionality. International standard. Switzerland: International Organization for Standardization; 2022.

[2]	ISO 26262: Road vehicles—functional safety. International standard. Switzerland: International Organization for Standardization; 2018.

[3]	N.G. Leveson. Engineering a safer world:systems thinking applied to safety. The MIT Press, Cambridge (2012)

[4]	P. Koopman, F. Fratrik. How many operational design domains, objects, and events?. Safeai@aaai, 4 (2019)

[5]	dmv.ca.gov Internet. Sacramento: California Department of Motor Vehicles; 2023 [cited 2023 Oct 26]. Available from:

[6]	S.A. Seshia, D. Sadigh, S.S. Sastry. Toward verified artificial intelligence. Commun ACM, 65 (7) (2022), pp. 46-55

[7]	Li J, Shao W, Wang H. Key challenges and Chinese solutions for SOTIF in intelligent connected vehicles. Engineering, in press.

[8]	S. Burton, I. Habli, T. Lawton, J. McDermid, P. Morgan, Z. Porter. Mind the gaps: assuring the safety of autonomous systems from an engineering, ethical, and legal perspective. Artif Intell, 279 (2020), Article 103201

[9]	J. Koo, J. Kwac, W. Ju, M. Steinert, L. Leifer, C. Nass. Why did my car just do that? Explaining semi-autonomous driving actions to improve driver understanding, trust, and performance. Int J Interact Des Manuf, 9 (4) (2015), pp. 269-275

[10]	K. Czarnecki, R. Salay. Towards a framework to manage perceptual uncertainty for safe automated driving. B. Gallina, A. Skavhaug, E. Schoitsch, F. Bitsch (Eds.), Computer Safety, Reliability, and Security, Springer, Berlin (2018), pp. 439-445

[11]	P. Koopman, U. Ferrell, F. Fratrik, M. Wagner. A safety standard approach for fully autonomous vehicles. A. Romanovsky, E. Troubitsyna, I. Gashi, E. Schoitsch (Eds.), Computer Safety, Reliability, and Security, Springer, Berlin (2019), pp. 326-332

[12]	UL 4600: Evaluation of autonomous products. UL standard. Underwriters Laboratories; 2020.

[13]	F. Concas, J.K. Nurminen, T. Mikkonen, S. Tarkoma. Validation frameworks for self-driving vehicles:a survey. M.A. Khan, F. Algarni, M.T. Quasim (Eds.), Smart Cities: A Data Analytics Perspective, Springer, Berlin (2021), pp. 197-212

[14]	ISO 34502: Road vehicles test scenarios for automated driving systems: scenario based safety evaluation framework. International standard. Switzerland: International Organization for Standardization; 2022.

[15]	Vesely WE, Goldberg FF, Roberts NH, Haasl DF. Fault tree handbook, systems and reliability research, Office of Nuclear Regulatory Research, US; 1981.

[16]

Schönemann V, Winner H, Glock T, Sax E, Boeddeker B, vom Dorff S, et al. Fault tree-based derivation of safety requirements for automated driving on the example of cooperative valet parking. In: 26th International Technical Conference on the Enhanced Safety of Vehicles (ESV); 2019 Jun 10-13; Eindhoven, Netherlands; 2019.

[17]	A. Börger, R. Hosse, S. Von Der Decken. SOTIF—a new challenge for functional testing. ATZelectronics Worldwide, 15 (10) (2020), pp. 56-60

[18]	J. Dunjó, V. Fthenakis, J.A. Vílchez, J. Arnaldos. Hazard and operability (HAZOP) analysis. A literature review. J Hazard Mater, 173 (1-3) (2010), pp. 19-32

[19]	Kramer B, Neurohr C, Büker M, Fränzle M, Damm W. Identification and quantification of hazardous scenarios for automated driving. In: Proceeding of Model-Based Safety and Assessment: 7th International Symposium; 2020 Sep 14-16; Lisbon, Portugal. Berlin: Springer; 2020. p. 163-78.

[20]	Song Y. Applying system-theoretic accident model and processes (STAMP) to hazard analysis [dissertation]. Hamilton: McMaster University; 2012.

[21]	N. Leveson. A new accident model for engineering safer systems. Saf Sci, 42 (4) (2004), pp. 237-270

[22]	Sundaram P, Vernacchia M, Wagner MS, Thomas J, Placke S. Application of STPA to an automotive shift-by-wire system. In: Workshop: Cambridge, MA, USA; 2014.

[23]	Van Eikema Hommes Q. Safety analysis approaches for automotive electronic control systems. In: Society of Automotive Engineers’ Meeting; 2015.

[24]	Van Eikema Hommes Q. Assessment of safety standards for automotive electronic control systems. Report. Washington, DC: National Highway Traffic Safety; 2016 Jun. Report No.: DOT HS 812 285.

[25]	A. Abdulkhaleq, S. Wagner, N. Leveson. A comprehensive safety engineering approach for software-intensive systems based on STPA. Procedia Eng, 128 (2015), pp. 2-11

[26]	A. Abdulkhaleq, D. Lammering, S. Wagner, J. Röder, N. Balbierer, L. Ramsauer, et al.. A systematic approach based on STPA for developing a dependable architecture for fully automated driving vehicles. Procedia Eng, 179 (2017), pp. 41-51

[27]	K. Czarnecki. On-road safety of automated driving system (ADS)—taxonomy and safety analysis methods. University of Waterloo, Waterloo (2018)

[28]	Xing X, Zhou T, Chen J, Xiong L, Yu Z. A hazard analysis approach based on STPA and finite state machine for autonomous vehicles. In: Proceeding of 2021 IEEE Intelligent Vehicles Symposium (IV); 2021 Jul 11-17; Nagoya, Japan. Piscataway: IEEE; 2006. p. 150-6.

[29]	C. Bensaci, Y. Zennir, D. Pomorski, F. Innal, Y. Liu, C. Tolba. STPA and Bowtie risk analysis study for centralized and hierarchical control architectures comparison. Alex Eng J, 59 (5) (2020), pp. 3799-3816

[30]	C. Bensaci, Y. Zennir, D. Pomorski, F. Innal, Y. Liu. Distributed vs. hybrid control architecture using STPA and AHP—application to an autonomous mobile multi-robot system. International Journal of Safety and Security. Engineering, 11 (1) (2021), pp. 1-12

[31]	Capito L, Redmill KA. Methodology for hazard identification and mitigation strategies applied to an overtaking assistant ADAS. In: Proceeding of 2021 IEEE International Intelligent Transportation Systems Conference (ITSC); 2021 Sep 19-22; Indianapolis, IN, USA. Piscataway: IEEE; 2021. p. 3972-7.

[32]	Zhao T, Yurtsever E, Paulson JA, Rizzoni G, Automated vehicle safety guarantee, verification and certification: a survey. 2022. arXiv:2202.02818v1.

[33]	J. Kapinski, J.V. Deshmukh, X. Jin, H. Ito, K. Butts. Simulation-based approaches for verification of embedded control systems: an overview of traditional and advanced modeling, testing, and verification techniques. IEEE Contr Syst Mag, 36 (6) (2016), pp. 45-64

[34]	Krook J, Svensson L, Li Y, Feng L, Fabian M. Design and formal verification of a safe stop supervisor for an automated vehicle. In: Proceeding of 2019 International Conference on Robotics and Automation (ICRA); 2019 May 20-24; Montreal, QC, Canada. Piscataway: IEEE; 5607-13.

[35]	C. Radojicic, C. Grimm, A. Jantsch, M. Rathmair. Towards verification of uncertain cyber-physical systems. Electron Proc Theor Comput Sci, 247 (2017), pp. 1-17

[36]	Arechiga N, Loos SM, Platzer A, Krogh BH. Using theorem provers to guarantee closed-loop system properties. In: Proceeding of 2012 American Control Conference (ACC); 2012 Jun 27-29; Montreal, QC, Canada. Piscataway: IEEE; 3573-80.

[37]	Gruber F, Althoff M. Anytime safety verification of autonomous vehicles. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC); 2018 Nov 4-7; Maui, HI, USA. Piscataway: IEEE; 2018. p. 1708-14.

[38]	B. Johnson, F. Havlak, H. Kress-Gazit, M. Campbell. Experimental evaluation and formal analysis of high-level tasks with dynamic obstacle anticipation on a full-sized autonomous vehicle. J Field Robot, 34 (5) (2017), pp. 897-911

[39]	Arechiga N. Specifying Safety of Autonomous vehicles in signal temporal logic. In: Proceeding of 2019 IEEE Intelligent Vehicles Symposium (IV); 2019 Jun 9-12; Paris. Piscataway: IEEE; 2018. p. 58-63.

[40]	E. Zapridou, E. Bartocci, P. Katsaros. Runtime verification of autonomous driving systems in CARLA. J. Deshmukh, D. Ničković (Eds.), Runtime Verification, Springer, Berlin (2020), pp. 172-183

[41]	Shalev-Shwartz S, Shammah S, Shashua A. On a formal model of safe and scalable self-driving cars. 2017. arXiv:1708.06374.

[42]	P. Nilsson, O. Hussien, A. Balkan, Y. Chen, A.D. Ames, J.W. Grizzle, et al.. Correct-by-construction adaptive cruise control: two approaches. IEEE Trans Control Syst Technol, 24 (4) (2016), pp. 1294-1307

[43]	N. Kalra, S.M. Paddock. Driving to safety: how many miles of driving would it take to demonstrate autonomous vehicle reliability?. Transp Res Part A Policy Pract, 94 (2016), pp. 182-193

[44]	S. Riedmaier, T. Ponn, D. Ludwig, B. Schick, F. Diermeyer. Survey on scenario-based safety assessment of automated vehicles. IEEE Access, 8 (2020), pp. 87456-87477

[45]

Ulbrich S, Menzel T, Reschka A, Schuldt F, Maurer M. Defining and substantiating the terms scene, situation, and scenario for automated driving. In: Proceeding of 2015 IEEE 18th International Conference on Intelligent Transportation Systems; 2015 Sep 15-18; Gran Canaria. Piscataway: IEEE; 2015. p. 982-8.

[46]	Bagschik G, Menzel T, Maurer M. Ontology based scene creation for the development of automated vehicles. In: 2018 IEEE Intelligent Vehicles Symposium (IV); 2018 Oct 21; Changshu, China. Piscataway: IEEE; 2018. p. 1813-20.

[47]	Khatun M, Glaß M, Jung R. A systematic approach of reduced scenario-based safety analysis for highly automated driving function. In: Proceedings of the 7th International Conference on Vehicle Technology and Intelligent Transport Systems; 2021 Apr 28; New York City, USA:2021. p. 301-8

[48]	W. Wang, D. Zhao. Extracting traffic primitives directly from naturalistically logged data for self-driving applications. IEEE Robot Autom Lett, 3 (2) (2018), pp. 1223-2129

[49]

Gladisch C, Heinzemann C, Herrmann M, Woehrle M. Leveraging combinatorial testing for safety-critical computer vision datasets. In: Proceeding of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW); 2020 Jun 14-19; Seattle, WA, USA; Piscataway. IEEE; 2018. p. 1314-21.

[50]	D. Zhao, X. Huang, H. Peng, H. Lam, D.J. LeBlanc. Accelerated evaluation of automated vehicles in car-following maneuvers. IEEE Trans Intell Transp Syst, 19 (3) (2018), pp. 733-744

[51]	S. Feng, X. Yan, H. Sun, Y. Feng, H.X. Liu. Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment. Nat Commun, 12 (1) (2021), p. 748

[52]	S. Feng, H. Sun, X. Yan, H. Zhu, Z. Zou, S. Shen, et al.. Dense reinforcement learning for safety validation of autonomous vehicles. Nature, 615 (7953) (2023), pp. 620-627

[53]	J. Wishart, S. Como, M. Elli, B. Russo, J. Weast, N. Altekar, et al.. Driving safety performance assessment metrics for ads-equipped vehicles. SAE Int J Adv Curr Prac Mobility, 2 (5) (2020), pp. 2881-2899

[54]	L.A. Dennis, M. Fisher. Verifiable self-aware agent-based autonomous systems. Proc IEEE, 108 (7) (2020), pp. 1011-1026

[55]

Gyllenhammar M, Johansson R, Warg F, Chen D, Heyn H M, Sanfridson M, et al. Towards an operational design domain that supports the safety argumentation of an automated driving system. In: Proceeding of 10th European Congress on Embedded Real Time Systems (ERTS 2020); 2020 Jan 29-31; Toulouse, France. 2020.

[56]	Colwell I, Phan B, Saleem S, Salay R, Czarnecki K. An automated vehicle safety concept based on runtime restriction of the operational design domain. In: Proceeding of 2018 IEEE Intelligent Vehicles Symposium (IV); 2018 Jun 26-30; Changshu, China; Piscataway. IEEE; 2018. p. 1910-7.

[57]	Reschka A, Bohmer JR, Saust F, Lichte B, Maurer M. Safe, dynamic and comfortable longitudinal control for an autonomous vehicle. In: Proceeding of 2012 IEEE Intelligent Vehicles Symposium; 2012 Jun 3-7; Madrid, Spain; Piscataway. IEEE; 2012. p. 346-51.

[58]

Reschka A, Böhmer JR, Nothdurft T, Hecker P, Lichte B, Maurer M. A surveillance and safety system based on performance criteria and functional degradation for an autonomous vehicle. In: Proceeding of 2012 15th International IEEE Conference on Intelligent Transportation Systems; 2012 Sep 16-19; Anchorage, AK, USA; Piscataway. IEEE; 2012. p. 237-42.

[59]	Schlatow J, Moostl M, Ernst R, Nolte M, Jatzkowski I, Maurer M, et al. Self-awareness in autonomous automotive systems. In: Proceeding of Design, Automation & Test in Europe Conference & Exhibition (DATE), 2017; 2017 Mar 27-31; Lausanne, Switzerland; Piscataway. IEEE; 2017. p. 1050-5.

[60]	Ulbrich S, Reschka A, Rieken J, Ernst S, Bagschik G, Dierkes F, et al. Towards a functional system architecture for automated vehicles. 2017. arXiv:2107.08142.

[61]	Nolte M, Jatzkowski I, Ernst S, Maurer M. Supporting safe decision making through holistic system-level representations & monitoring—a summary and taxonomy of self-representation concepts for automated vehicles. 2020. arXiv:2007.13807.

[62]	W. Shao, J. Li, Y. Zhang, H. Wang. [Key technologies to ensure the safety of the intended functionality for intelligent vehicles]. Automot Eng, 44 (2022), pp. 1289-1304Chinese

[63]	Jain A, Del Pero L, Grimmett H, Ondruska P. Autonomy 2.0: why is self-driving always 5 years away? 2021. arXiv:2107.08142.

[64]	ISO/TR 4804: Road vehicles—safety and cybersecurity for automated driving systems: design, verification and validation. International standard. Switzerland: International Organization for Standardization, 2020.

[65]	Kuang X, Zhang Y, Li H. SOTIF requirement analysis based on STPA. In: Proceedings of the 2021 4th International Conference on Algorithms, Computing and Artificial Intelligence; 2021 Dec 22-24; Sanya, China; New York City: ACM Digital Library. 2021. p. 1-5.

[66]	H.H. Goode, R.E. Machol, T. Teichmann. System engineering. Phys Today, 10 (9) (1957), pp. 34-36

[67]	Xu Y, Shao W, Li J, Yang K, Wang W, Huang H, et al. SIND: A drone dataset at signalized intersection in China. In: Proceeding of 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC); 2022 Oct 8-12; Macao, China; Piscataway: IEEE; 2022. p. 2471-8.

[68]	J. Wang, J. Wu, X. Zheng, D. Ni, K. Li. Driving safety field theory modeling and its application in pre-collision warning system. Transp Res, Part C Emerg Technol, 72 (2016), pp. 306-324

[69]	Hendrycks D, Basart S, Mu N, Kadavath S, Wang F, Dorundo E, et al. The many faces of robustness: a critical analysis of out-of-distribution generalization. 2021. arXiv.2006.16241.

[70]	Shafahi A, Najibi M, Ghiasi A, Xu Z, Dickerson J, Studer C, et al. Adversarial training for free!In: Proceedings of the 33rd International Conference on Neural Information Processing Systems; 2019 Dec 8-14; Red Hook, NY, USA; New York Ctiy: ACM Digital Library; 2019. p. 3358-69.

[71]	F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, et al.. A comprehensive survey on transfer learning. Proc IEEE, 109 (1) (2021), pp. 43-76

[72]

R. Luo, S. Zhao, J. Kuck, B. Ivanovic, S. Savarese, E. Schmerling, et al.. Sample-efficient safety assurances using conformal prediction. Proceeding of, S.M.LaValle, J.M.O’Kane, M.Otte, D.Sadigh, P.Tokekar (Eds.), Algorithmic foundations of robotics XV, Springer International Publishing, Springer, Berlin (2023), pp. 149-169

[73]

Sadat A, Ren M, Pokrovsky A, Lin YC, Yumer E, Urtasun R. Jointly learnable behavior and trajectory planning for self-driving vehicles. In: Proceeding of 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); 2019 Nov 1 Macao, China. New York City: ACM Digital Library; 3949-56.

[74]	Ross S, Bagnell D. Efficient Reductions for Imitation Learning. In: Teh, Y.W., Titterington, M., editors. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics; 2010 May 13-15; Sardinia, Italy. Pittsburgh: PMLR; 2010. p. 661-8.

[75]	Gansch R, Adee A. System theoretic view on uncertainties. In: Proceeding of 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE); 2020 Mar 9-13; Grenoble, France; Piscataway: IEEE; 2022. p. 1345-50.

[76]

Adee A, Munk P, Gansch R, Liggesmeyer P. Uncertainty representation with extended evidential networks for modeling safety of the intended functionality (SOTIF). In:Proceedings of the 30th European Safety and Reliability Conference and 15th Probabilistic Safety Assessment and Management Conference; 2020 Nov 1-5; Venice, Italy; 2020. p. 4148-55.

[77]	W. Shao, Y. Xu, J. Li, C. Lv, W. Wang, H. Wang. How does traffic environment quantitatively affect the autonomous driving prediction?. IEEE Trans Intell Transp Syst, 24 (10) (2023), pp. 11238-11253

[78]	E. Hüllermeier, W. Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Mach Learn, 110 (3) (2021), pp. 457-506

[79]	K. Yang, X. Tang, J. Li, H. Wang, G. Zhong, J. Chen, et al.. Uncertainties in onboard algorithms for autonomous vehicles: challenges, mitigation, and perspectives. IEEE Trans Intell Transp Syst, 24 (9) (2023), pp. 8963-8987

[80]	Kendall A, Gal Y. What uncertainties do we need in Bayesian deep learning for computer vision? In: Proceedings of the 31st International Conference on Neural Information Processing Systems; 2017 Dec 4-9; Red Hook, NY, USA; New York City: ACM Digital Library; 2017.

[81]	G. Montavon, W. Samek, K.R. Müller. Methods for interpreting and understanding deep neural networks. Digit Signal Process, 73 (2018), pp. 1-15

[82]	Y. Zhang, P. Tino, A. Leonardis, K. Tang. A survey on neural network interpretability. IEEE Trans Emerg Top Comput Intell, 5 (5) (2021), pp. 726-742

[83]	W.J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, B. Yu. Definitions, methods, and applications in interpretable machine learning. Proc Natl Acad Sci USA, 116 (44) (2019), pp. 22071-22080

[84]	Louizos C, Welling M. Multiplicative normalizing flows for variational Bayesian neural networks. In: Proceedings of the 34th International Conference on Machine Learning; 2017 Aug 6-11; Sydney, Australia; New York City: ACM Digital Library; 2017. p. 2218-27.

[85]	Kristiadi A, Hein M, Hennig P. Learnable uncertainty under laplace approximations. 2020. arXiv:2010.02720.

[86]	Salimans T, Kingma DP, Welling M. Markov Chain Monte Carlo and variational inference: bridging the gap. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning—Volume 37; 2015 Jul 7-9; Lille, France; 2015. p. 1218-26.

[87]	Gal Y, Ghahramani Z. Dropout as a bayesian approximation:representing model uncertainty in deep learning. In: Proceedings of the 33rd International Conference on Machine Learning; 2016 Jun 19-24; New York, NY, USA. New York City: ACM Digital Library; 2016. p. 1050-9.

[88]	Lakshminarayanan B, Pritzel A, Blundell C. Simple and scalable predictive uncertainty estimation using deep ensembles. In: Proceedings of the 31st International Conference on Neural Information Processing Systems; 2017 Dec 4-9; Red Hook, NY, USA. New York City: ACM Digital Library; 2017.

[89]	Wenzel F, Snoek J, Tran D, Jenatton R. Hyperparameter ensembles for robustness and uncertainty quantification. In: Proceedings of the 34th International Conference on Neural Information Processing Systems; 2020 Dec 6-12; Red Hook, NY, USA. New York City: ACM Digital Library; 2020. p. 6514-27.

[90]	Wen Y, Tran D, Ba J. BatchEnsemble: an alternative approach to efficient ensemble and lifelong learning. 2020. arXiv:2002.06715.

[91]	Malinin A, Gales M. Predictive uncertainty estimation via prior networks. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems; 2018 Dec 3-8; Red Hook, NY, USA. New York City: ACM Digital Library; 2018.

[92]	Sensoy M, Kaplan L, Kandemir M. Evidential deep learning to quantify classification uncertainty. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems; 2018 Dec 3-8; Red Hook, NY, USA. New York City: ACM Digital Library; 2018.

[93]	Rosenfeld N, Mansour Y, Yom-Tov E. Discriminative learning of prediction intervals. In: Storkey A, Perez-Cruz F, editors. In:Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics; 2018 Apr 9- 11; Lanzarote, Spain. Pittsburgh: PMLR; 2018. p. 347-55.

[94]	Chang J, Lan Z, Cheng C, Wei Y. Data uncertainty learning in face recognition. In: Proceeding of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2020 Jun 14-19; Seattle, WA, USA. Piscataway: IEEE. 2020. p. 5709-18.

[95]	Y. Gao, M.K. Ng. Wasserstein generative adversarial uncertainty quantification in physics-informed neural networks. J Comput Phys, 463 (2022), Article 111270

[96]	M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, et al.. A review of uncertainty quantification in deep learning: techniques, applications and challenges. Inf Fusion, 76 (2021), pp. 243-297

[97]	D. Feng, A. Harakeh, S.L. Waslander, K. Dietmayer. A review and comparative study on probabilistic object detection in autonomous driving. IEEE Trans Intell Transp Syst, 23 (8) (2022), pp. 9961-9980

[98]	M. Du, N. Liu, X. Hu. Techniques for interpretable machine learning. Commun ACM, 63 (1) (2019), pp. 68-77

[99]	D.V. Carvalho, E.M. Pereira, J.S. Cardoso. Machine learning interpretability: a survey on methods and metrics. Electronics, 8 (8) (2019), p. 832

[100]

J. Janai, F. Güney, A. Behl, A. Geiger. Computer vision for autonomous vehicles: problems, datasets and state of the art. Found Trends Comput Graph Vis (2017) arXiv:1704.05519

[101]

O. Willers, S. Sudholt, S. Raafatnia, S. Abrecht. Safety concerns and mitigation approaches regarding the use of deep learning in safety-critical perception tasks. A. Casimiro, F. Ortmeier, E. Schoitsch, F. Bitsch, P. Ferreira (Eds.), Computer safety, reliability, and security, Springer, Berlin (2020), pp. 336-350

[102]

Alcorn MA, Li Q, Gong Z, Wang C, Mai L, Ku WS, et al. Strike (with) a pose: neural networks are easily fooled by strange poses of familiar objects. In: Proceeding of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2019 Jun 15-20; Long Beach, CA, USA. Piscataway: IEEE. 2019. p. 4840-9.

[103]

Remeli V, Morapitiye S, Rovid A, Szalay Z. Towards verifiable specifications for neural networks in autonomous driving. In:Proceeding of 2019 IEEE 19th International Symposium on Computational Intelligence and Informatics and 7th IEEE International Conference on Recent Achievements in Mechatronics, Automation, Computer Sciences and Robotics (CINTI-MACRo); 2019 Nov 14- 16 ; Szeged, Hungary. Piscataway: IEEE. 2019. p. 175-80.

[104]

Sämann T, Schlicht P, Hüger Strategy to increase the safety of a DNN-based perception for HAD systems. 2020. arXiv:2002.08935.

[105]

J. Hariyono, K.H. Jo. Detection of pedestrian crossing road: a study on pedestrian pose recognition. Neurocomputing, 234 (2017), pp. 144-153

[106]

Ajanovic Z, Lacevic B, Shyrokau B, Stolz M, Horn M. Search-based optimal motion planning for automated driving. In: Proceeding of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); 2018 Oct 1-5; Madrid, Spain. Piscataway: IEEE. 2018. p. 4523-30.

[107]

J. Hu, B. Zheng, C. Wang, C. Zhao, X. Hou, Q. Pan, et al.. A survey on multi-sensor fusion based obstacle detection for intelligent ground vehicles in off-road environments. Front Inf Technol Electron Eng, 21 (5) (2020), pp. 675-692

[108]

Zhao X, Liu Z, Hu R, Huang K. 3D object detection using scale invariant and feature reweighting networks.Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence; 2019 Jan 27-Feb 1; Honolulu, HI, New York City: ACM Digital Library. 2019. p. 9267-74.

[109]

L. Guan, Y. Chen, G. Wang, X. Lei. Real-time vehicle detection framework based on the fusion of lidar and camera. Electronics, 9 (3) (2020), p. 451

[110]

Xu R, Xiang H, Xia X, Han X, Li J, Ma J. OPV2V:an open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. In: Proceeding of 2022 International Conference on Robotics and Automation (ICRA); 2022 May 23-27; Philadelphia, PA, USA. Piscataway: IEEE; 2022. p. 2583-9.

[111]

S. Khan, F. Andert, N. Wojke, J. Schindler, A. Correa, A. Wijbenga. Towards collaborative perception for automated vehicles in heterogeneous traffic. J. Dubbert, B. Müller, G. Meyer (Eds.), Advanced microsystems for automotive applications 2018, Springer, Berlin (2019), pp. 31-42

[112]

S. Aradi. Survey of deep reinforcement learning for motion planning of autonomous vehicles. IEEE Trans Intell Transp Syst, 23 (2) (2022), pp. 740-759

[113]

Z. Zhu, H. Zhao. A survey of deep RL and IL for autonomous driving policy learning. IEEE Trans Intell Transp Syst, 23 (9) (2022), pp. 14043-14065

[114]

Xu J, Shao W, Xu Y, Wang W, Li J, Wang H. A risk probability predictor for effective downstream planning tasks. In: Proceeding of 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC); 2023 Sep 24-28; Bilbao, Spain. Piscataway: IEEE; 2023.

[115]

X. Wang, X. Qi, P. Wang, J. Yang. Decision making framework for autonomous vehicles driving behavior in complex scenarios via hierarchical state machine. Autonomous Intel Syst, 1 (1) (2021), p. 10

[116]

Orzechowski PF, Burger C, Lauer M. Decision-making for automated vehicles using a hierarchical behavior-based arbitration scheme. In: Proceeding of 2020 IEEE Intelligent Vehicles Symposium (IV); 2020 Oct 19-Nov 13; Las Vegas, NV, USA. Piscataway: IEEE; 2020. p. 767-74.

[117]

P. Hang, S. Huang, X. Chen, K.K. Tan. Path planning of collision avoidance for unmanned ground vehicles: a nonlinear model predictive control approach. Proc Inst Mech Eng, Part I, J Syst Control Eng, 235 (2) (2021), pp. 222-236

[118]

X. Zhang, W. Shao, M. Zhou, Q. Tan, J. Li. A scene comprehensive safety evaluation method based on binocular camera. Robot Auton Syst, 128 (2020), Article 103503

[119]

Zhao S, Hou Q, Zhai Y. Decision mechanism of vehicle autonomous lane change based on rough set theory. In: Proceedings of the 2021 1st International Conference on Control and Intelligent Robotics; 2021 Jun 18-20; New York, NY, USA. New York City: ACM Digital Library; 2021. p. 33-9.

[120]

Beheshtitabar E, Mohammad Alipour E. A rule based control algorithm for on-ramp merge with connected and automated vehicles. In: Proceeding of International Conference on Transportation and Development 2020; 2020 May 26-29; Washington, DC, USA. Washington, DC: American Society of Civil Engineers; 2020. p. 303-16.

[121]

B.R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A.A.A. Sallab, S. Yogamani, et al.. Deep reinforcement learning for autonomous driving: a survey. IEEE Trans Intell Transp Syst, 23 (6) (2022), pp. 4909-4926

[122]

H. Gao, G. Shi, G. Xie, B. Cheng. Car-following method based on inverse reinforcement learning for autonomous vehicle decision-making. Int J Adv Robot Syst, 15 (6) (2018), pp. 1-11

[123]

Brown A, Petrik, Interpretable reinforcement learning with ensemble methods; 2018. arXiv:1809.06995.

[124]

Moldovan TM, Abbeel P. Safe exploration in Markov decision processes; 2012.arXiv:1205.4810.

[125]

Nishimura H, Ivanovic B, Gaidon A, Pavone M, Schwager M. Risk-sensitive sequential action control with multi-modal human trajectory forecasting for safe crowd-robot interaction. In: Proceeding of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); 2020 Oct 24; Las Vegas, NV, USA. New York City: ACM Digital Library. 2020. p. 11205-12.

[126]

Z. Cao, S. Xu, X. Jiao, H. Peng, D. Yang. Trustworthy safety improvement for autonomous driving using reinforcement learning. Transp Res, Part C Emerg Technol, 138 (2022), Article 103656

[127]

C. Pek, S. Manzinger, M. Koschi, M. Althoff. Using online verification to prevent autonomous vehicles from causing accidents. Nat Mach Intell, 2 (9) (2020), pp. 518-528

[128]

Z. Cao, K. Jiang, W. Zhou, S. Xu, H. Peng, D. Yang. Continuous improvement of self-driving cars using dynamic confidence-aware reinforcement learning. Nat Mach Intell, 5 (2) (2023), pp. 145-158

[129]

Urban C, Miné A. A review of formal methods applied to machine learning. 2021. arXiv.2104.02466.

[130]

E.M. Clarke, T.A. Henzinger, H. Veith, R. Bloem (Eds.), Handbook of model checking, Springer, Berlin (2018)

[131]

G. Katz, D.A. Huang, D. Ibeling, K. Julian, C. Lazarus, R. Lim, et al.. The marabou framework for verification and analysis of deep neural networks. I.Dillig, S.Tasiran (Eds.), Computer aided verification, Springer, Berlin (2019), pp. 443-452

[132]

R.R. Bunel, I. Turkaslan, P.H.S. Torr, P. Kohli, P.K. Mudigonda Proceeding of A unified view of piecewise linear neural network verification, ACM Digital Library, Red Hook, NY, USA. New York (2018), pp. 4795-4804

[133]

G. Singh, T. Gehr, M. Püschel, M. Vechev. An abstract domain for certifying neural networks. Proc ACM Program Lang, 3 (2019), p. 41

[134]

X. Huang, M. Kwiatkowska, S. Wang, M. Wu. Safety verification of deep neural networks. R. Majumdar, V. Kunčak (Eds.), Computer aided verification, Springer, Berlin (2017), pp. 3-29

[135]

Weng TW, Zhang H, Chen PY, Yi J, Su D, Gao Y, et al., Evaluating the robustness of neural networks: an extreme value theory approach. 2018. arXiv:1801.10578.

[136]

Lee S, Cha S, Lee D, Oh H. Effective white-box testing of deep neural networks with adaptive neuron-selection strategy. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis; 2020 Jul 18-22; New York, USA. New York City: ACM Digital Library; 2020. p. 165-76.

[137]

Byun T, Rayadurgam S, Heimdahl MPE. Black-box testing of deep neural networks. In: Proceeding of 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE); 2021 Oct 25-28; Wuhan, China. Piscataway: IEEE; 2021. p. 309-20.

[138]

Ma L, Zhang F, Sun J, Xue M, Li B, Juefei-Xu F, et al. DeepMutation: mutation testing of deep learning systems. In: Proceeding of 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE); 2018 Oct 15-18; Memphis, TN, USA. Piscataway: IEEE; 2018. p. 100-11.

[139]

Z.Q. Zhou, L. Sun. Metamorphic testing of driverless cars. Commun ACM, 62 (3) (2019), pp. 61-67

[140]

Ma L, Juefei-Xu F, Zhang F, Sun J, Xue M, Li B, et al. DeepGauge:multi-granularity testing criteria for deep learning systems. In:Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering; 2018 Sep 3-7; Montpellier, France. Piscataway: IEEE; 2018. p. 120-31.

[141]

Aggarwal A, Shaikh S, Hans S, Haldar S, Ananthanarayanan R., Saha D. Testing framework for black-box AI models. In:Proceeding of 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion); 2021 May 25-25; Madrid, Spain. Piscataway: IEEE; 2021. p. 81-4.

[142]

Volk G, Gamerdinger J, von Bernuth A, Bringmann O. A comprehensive safety metric to evaluate perception in autonomous systems. In: Proceeding of 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC); 2020 Sep 20-23; Rhodes, Greece. Piscataway: IEEE; 2020. p. 1-8.

[143]

Ivanovic B., Pavone M., Rethinking trajectory forecasting evaluation; 2021. arXiv:2106.12732.

[144]

B.L. Deng, G. Li, S. Han, L. Shi, Y. Xie. Model compression and hardware acceleration for neural networks: a comprehensive survey. Proc IEEE, 108 (4) (2020), pp. 485-532

[145]

V. Sze, Y.H. Chen, T.J. Yang, J.S. Emer. Efficient processing of deep neural networks: a tutorial and survey. Proc IEEE, 105 (12) (2017), pp. 2295-2329

[146]

Wei T, Liu C. Online Verification of deep neural networks under domain shift or network updates; 2023. arXiv:2106.12732.

[147]

F. Zhao, C. Zhang, N. Dong, Z. You, Z. Wu. A uniform framework for anomaly detection in deep neural networks. Neural Process Lett, 54 (4) (2022), pp. 3467-3488

[148]

Shao W, Li B, Yu W, Xu J, Wang H. When is it likely to fail? Performance monitor for black-box trajectory prediction model. 2023. techrxiv.24265672.v1.

[149]

Mougan C, Nielsen DS. Monitoring model deterioration with explainable uncertainty estimation via non-parametric bootstrap. In:Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence; 2023 Feb 7- 14 ; New York City: ACM Digital Library; 2023. p. 15037-45.

[150]

Q.M. Rahman, P. Corke, F. Dayoub. Run-time monitoring of machine learning for robotic perception: a survey of emerging trends. IEEE Access, 9 (2021), pp. 20067-20075

[151]

Mohseni S, Pitale M, Singh V, Wang Z. Practical solutions for machine learning safety in autonomous vehicles. 2019. arXiv:1912.09630.

[152]

Mohseni S, Jagadeesh A, Wang Z. Predicting model failure using saliency maps in autonomous driving systems. 2019. arXiv:1905.07679.

[153]

Hecker S, Dai D, Van Gool L. Failure prediction for autonomous driving. In: Proceeding of 2018 IEEE Intelligent Vehicles Symposium (IV); 2018 Jun 26-30; Changshu, China. Piscataway: IEEE; 2018. p. 1792-9.

[154]

C. Gurău, D. Rao, C.H. Tong, I. Posner. Learn from experience: probabilistic prediction of perception performance to avoid failure. Int J Robot Res, 37 (9) (2018), pp. 981-995

[155]

J. Yang, S. Rahardja, P. Fränti. Mean-shift outlier detection and filtering. Pattern Recognit, 115 (2021), Article 107874

[156]

Grathwohl W, Wang KC, Jacobsen JH, Duvenaud D, Norouzi M, Swersky K. Your classifier is secretly an energy based model and you should treat it like one. 2020. arXiv:1912.03263.

[157]

C. Gautam, R. Balaji, A. Tiwari, K. Ahuja. Localized multiple kernel learning for anomaly detection: one-class classification. Knowl Base Syst, 165 (2019), pp. 241-252

[158]

Gu X, Akoglu L, Rinaldo A. Statistical analysis of nearest neighbor methods for anomaly detection. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems; 2018 Dec 8; Red Hook, NY, USA. New York City: ACM Digital Library; 2019. p. 10923-33.

[159]

Sharan V, Gopalan P, Wieder U. Efficient anomaly detection via matrix sketching. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems; 2018 Dec 3-8; Red Hook, NY, USA. New York City: ACM Digital Library; 2018. p. 8080-91.

[160]

Chalapathy R, Chawla S. Deep learning for anomaly detection: a survey. 2019. arXiv:1901.03407.

[161]

B. Lindemann, B. Maschler, N. Sahlab, M. Weyrich. A survey on anomaly detection for technical systems using LSTM networks. Comput Ind, 131 (2021), Article 103498

[162]

Kim KH, Shim S, Lim Y, Jeon J, Choi J, Kim B, et al. RaPP: novelty detection with reconstruction along projection pathway. In:Proceedings of the International Conference on Learning Representations (ICLR) 2020; 2020; Virtual Conference.

[163]

Goodfellow J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al., Generative adversarial networks. 2014. arXiv:1406.2661.

[164]

D.P. Kingma, M. Welling. An introduction to variational autoencoders. Found Trends Mach Learn, 12 (4) (2019), pp. 307-392

[165]

Corbière C, Thome N, Bar-Hen A, Cord M, Pérez P. Addressing failure prediction by learning model confidence. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems; 2018 Dec 8; Red Hook, NY, USA. New York City: ACM Digital Library; 2019. p. 2902-2913.

[166]

Shao W, Li J, Wang H. Self-aware trajectory prediction for safe autonomous driving. In: Proceeding of 2023 IEEE Intelligent Vehicles Symposium (IV); 2023 Jun 4-7; Anchorage, AK, USA. Piscataway: IEEE; 2023. p. 1-8.

[167]

Rahman QM, Sunderhauf N, Dayoub F. Online monitoring of object detection performance during deployment. In: Proceeding of 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); 2021 Sep 27-Oct 1; Prague, Czech Republic. Piscataway: IEEE; 4839-45.

[168]

Rahman QM, Sunderhauf N, Dayoub F. Per-frame mAP Prediction for continuous performance monitoring of object detection during deployment. In: 2021 IEEE Winter Conference on Applications of Computer Vision Workshops (WACVW); 2021 Jan 5-9; Waikola, HI, USA. Piscataway: IEEE; 2021. p. 152-60.

[169]

Henzinger TA, Lukina A, Schilling C. Outside the box: abstraction-based monitoring of neural networks. 2020. arXiv:1911.09032.

[170]

S. Luan, Z. Gu, L.B. Freidovich, L. Jiang, Q. Zhao. Out-of-distribution detection for deep neural networks with isolation forest and local outlier factor. IEEE Access, 9 (2021), pp. 132980-132989

[171]

Lee K, Lee K, Lee H, Shin J. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems; 2018 Dec 3-8; Red Hook, NY, USA. New York City: ACM Digital Library; 2018.

[172]

Huang R, Geng A, Li Y. On the importance of gradients for detecting distributional shifts in the wild. In:Proceedings of the Advances in Neural Information Processing Systems 34 (NeurIPS 2021); 2021 Dec; Virtual Conference.

[173]

Hendrycks D, Gimpel K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In:Proceedings of International Conference on Learning Representations; 2017.

[174]

Liang S, Li Y, Srikant R. Enhancing the reliability of out-of-distribution image detection in neural networks. 2018. arXiv:1706.02690.

[175]

Shao W, Xu Y, Peng L, Li J, Li J, Wang H. Failure detection for motion prediction of autonomous driving: an uncertainty perspective. In: Proceeding of 2023 IEEE International Conference on Robotics and Automation (ICRA); 2023 May 29-Jun 2; London, UK. Piscataway: IEEE; 2023. p. 12721-8.

[176]

K. Yang, B. Li, W. Shao, X. Tang, X. Liu, H. Wang. Prediction failure risk-aware decision-making for autonomous vehicles on signalized intersections. IEEE Trans Intell Transp Syst, 24 (11) (2023), pp. 12806-12820

[177]

Kaur R, Jha S, Roy A, Park S, Sokolsky O, Lee I. Detecting OODs as datapoints with high uncertainty; 2021. arXiv:2108.06380.

[178]

Weiss M, Tonella P. Fail-safe execution of deep learning based systems through uncertainty monitoring. In: Proceeding of 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST); 2021 Apr 12-16; Porto de Galinhas, Brazil. Piscataway: IEEE; 2021. p. 24-35.

[179]

M.S. Ramanagopal, C. Anderson, R. Vasudevan, M. Johnson-Roberson. Failing to learn: autonomously identifying perception failures for self-driving cars. IEEE Robot Autom Lett, 3 (4) (2018), pp. 3860-3867

[180]

Antonante P, Spivak DI, Carlone L. Monitoring and diagnosability of perception systems. In: Proceeding of 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); 2021 Sep 27-Oct 1; Prague, Czech Republic. Piscataway: IEEE; 168-75.

[181]

T. Stahl, F. Diermeyer. Online verification enabling approval of driving functions—implementation for a planner of an autonomous race vehicle. IEEE Open J Intell Transp Syst, 2 (2021), pp. 97-110

[182]

BMW Group. Safety assessment report: SAE Level 3 automated driving system. [Internet]. 2022 [cited 2023 Oct 26]. Available from:

[183]

Baidu, Apollo pilot safety report. [Internet]. 2018 [cited 2023 Oct 26]. Available from:

[184]

GM. Self-driving safety report [Internet]. 2018 [cited 2023 Oct 26]. Available from:

[185]

Ford Motor Company. A matter of trust: ford releases safety assessment report for self-driving vehicle development. [Internet]. Dearborn, MI: Business Wire; 2018 Aug 16 [cited 2023 Oct 26]. Available from:

[186]

Nuro. Delivering safety: Nuro VSSA Dec 2021 [Internet]. 2021 Dec 16 [cited 2023 Oct 26]. Available from:

[187]

NAVYA. Safety report [Internet]. 2019 [cited 2023 Oct 26]. Available from:

[188]

NVIDIA. Self-driving safety report 2018 [Internet]. 2018 [cited 2023 Oct 26]. Available from:

[189]

Ebel S., Bosch case study: application of SOTIF for ADAS. Report. Robert Bosch GmbH 2018, n.d.

[190]

APTIV, AUDI, BAIDU, BMW, Continental, FCA, et al. Safety first for automated driving [Internet]. 2019 [cited 2023 Oct 26]. Available from:

[191]

Kaiser B. An integrative solution towards SOTIF and AV safety. In: Proceeding of IQPC SOTIF Conference; 2019 Oct 1-2; Austin, TX, USA; 2019 ; Austin, TX, USA; 2019.

[192]

Becker C, Brewer JC, Yount L, John A. Safety of the intended functionality of lane-centering and lane-changing maneuvers of a generic level 3 highway chauffeur system. Report. Washington D. C.: National Highway Traffic Safety Administration; 2020. Report No.: DOT HS 812 879.

[193]

hella.com [Internet]. Lippstadt: HELLA GmbH & Co. KGaA; [cited 2023 Oct 26]. Available from:

[194]

Mobileye Mobileye safety methodology. Report. Mobileye; 2023.

[195]

Junietz P, Wachenfeld W, Klonecki K, Winner H. Evaluation of different approaches to address safety validation of automated driving. In: Proceeding of 2018 21st International Conference on Intelligent Transportation Systems (ITSC); 2018 Nov 4-7; Maui, HI, USA. Piscataway: IEEE; 2018. p. 491-6.

[196]

S. Hallerbach, Y. Xia, U. Eberle, F. Koester. Simulation-based identification of critical scenarios for cooperative and automated vehicles. SAE Int J Connect Autom Veh, 1 (2) (2018), pp. 93-106

[197]

Holder M, Rosenberger P, Winner H, Dhondt T, Makkapati VP, Maier M, et al. Measurements revealing challenges in radar sensor modeling for virtual validation of autonomous driving. In: Proceeding of 2018 21st International Conference on Intelligent Transportation Systems (ITSC); 2018 Nov 4-7; Maui, HI, USA. Piscataway: IEEE; 2018. p. 2616-22.

[198]

Duy Son T, Bhave A, Van Der Auweraer H. Simulation-based testing framework for autonomous driving development. In: Proceeding of 2019 IEEE International Conference on Mechatronics (ICM); 2019 Mar 18-20; Ilmenau, Germany. Piscataway: IEEE; 2019. p. 576-83.

[199]

Siemens AG. Scenario-based validation and verification of automated driving systems [Internet]. Siemens AG; 2022 [cite 2023 Oct 26]. Available from:

[200]

Akagi Y, Kato R, Kitajima S, Antona-Makoshi J, Uchida N. A risk-index based sampling method to generate scenarios for the evaluation of automated driving vehicle safety. In: Proceeding of 2019 IEEE Intelligent Transportation Systems Conference (ITSC); 2019 Oct 27-30; Auckland, New Zealand. Piscataway: IEEE; 2019. p. 667-72.

[201]

H. Nakamura, H. Muslim, R. Kato, S. Préfontaine-Watanabe, H. Nakamura, H. Kaneko, et al.. Defining reasonably foreseeable vehicle parameter ranges for scenario-based testing of automated vehicles in consideration of risk acceptance. IEEE Access, 10 (2021), pp. 37743-37760

[202]

Thal S, Znamiec H, Henze R, Nakamura H, Imanaga H, Antona-Makoshi J, et al. Incorporating safety relevance and realistic parameter combinations in test-case generation for automated driving safety assessment. In: Proceeding of 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC); 2020 Sep 20-23; Rhodes, Greece. Piscataway: IEEE; 2020. p. 1-6.

[203]

Economic Commission for Europe. UN Regulation No. 157 (Automated Lane Keeping Systems) [Internet]. Geneva: United Nations Economic Commission for Europe; 2022 Mar 5 [cited 2023 Oct 26]. Available from:

[204]

Japan Automobile Manufacturers Association, Inc. Automated Driving Safety Evaluation Framework Ver 3.0 [Internet]. Tokyo:Japan Automobile Manufacturers Association, Inc.; 2022 Dec. Available from:

[205]

Borg M, Englund C, Wnuk K, Duran B, Levandowski C, Gao S, et al. Safely entering the deep: a review of verification and validation for machine learning and a challenge elicitation in the automotive industry. 2018. arXiv:1812.05389.

[206]

Henriksson J, Borg M, Englund C. Automotive safety and machine learning:initial results from a study on how to adapt the ISO 26262 safety standard. In:Proceedings of the 1st International Workshop on Software Engineering for AI in Autonomous Systems; 2018 May 28; Gothenburg, Sweden. Piscataway: IEEE; 2018. p. 47-49.

[207]

Henriksson J, Berger C, Borg M, Tornberg L, Englund C, Sathyamoorthy SR, et al. Towards structured evaluation of deep neural network supervisors. In: Proceeding of 2019 IEEE International Conference On Artificial Intelligence Testing (AITest); 2019 Apr 4-9, Newark, CA, USA. Piscataway: IEEE; 2019. p. 27-34.

[208]

IVEX NV. Safety Co-pilot [Internet]. Heverlee: IVEX NV; 2023 [cited 2023 Oct 26]. Available from:

[209]

PAS 1880:2020: Guidelines for developing and assessing control systems for automated vehicles. British Standards Institution; 2020.

[210]

Karpathy A. Multi-Task Learning in the Wilderness [Internet]. Long Beach, CA: SlidesLive; 2019 Jun 15 [cited 2023 Oct 26]. Available from:

[211]

Gao P. You Should Try Active Learning! [Internet]. Medium; 2021 Jan 28 [cited 2023 Oct 26]. Available from:

[212]

Koumchatzky N. Maglev:software 2.0 platform for autonomous vehicles development. Report. Santa Clara: NVIDIA; 2020.

[213]

Motional. Technically speaking:learning with every mile driven. Report. Boston: Motional; 2021.

[214]

Harris S. Cruise’s continuous learning machine predicts the unpredictable on San Francisco roads [Internet]. Medium; 2020 Sep 11 [cited 2023 Oct 26]. Available from:

[215]

H.S. Mahajan, T. Bradley, S. Pasricha. Application of systems theoretic process analysis to a lane keeping assist system. Reliab Eng Syst Saf, 167 (2017), pp. 177-183

[216]

Stolte T, Bagschik G, Maurer M. Safety goals and functional safety requirements for actuation systems of automated vehicles. In: Proceeding of 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC); 2016 Nov 1-4; Rio de Janeiro, Brazil. Piscataway: IEEE; 2016. p. 2191-8.

[217]

Zhao S, Duan J, Wu S, Gu X, Li C, Yin K, et al. Genetic algorithm-based SOTIF scenario construction for complex traffic flow. Automot Innov 2023;6:531-46.

[218]

P Cao, W Wachenfeld, H Winner. Perception sensor modeling for virtual validation of automated driving. It-Information Technology, 57 (2015), pp. 243-251

[219]

Linder A, Davidse RJ, Iraeus J, John J, Keller A, Klug C, et al. VIRTUAL—a European approach to foster the uptake of virtual testing in vehicle safety assessment. In: Proceedings of 8th Transport Research Arena TRA 2020; 2020 Apr 27-30; Helsinki, Finland. 2020.

[220]

Yahoo. Honda wins world-first approval for level 3 autonomous car. Report. Science X (2020)

[221]

Capperella. Mercedes drive pilot level 3 autonomous system to launch in Germany. Report, Car and Driver, Harlan (2023)

[222]

Proposal for a new UN Regulation on: uniform provisions concerning the approval of vehicles with regard to Automated Lane Keeping Systems. Report. 2021 Oct. Report No.: GRSG-122-16.

[223]

Madala K, Krishnamoorthy J, Gonzalez CA, Shivkumar A, Solmaz M. Contributing factors to consider while defining acceptance criteria and validation targets for assuring SOTIF in autonomous vehicles, SAE Technical Paper 2022.

[224]

Favaro F, Fraade-Blanar L, Schnelle S, Victor T, Peña M, Engstrom J, et al., Building a credible case for safety: waymo’s approach for the determination of absence of unreasonable risk. 2023. arXiv:2306.01917.

[225]

B Boddeker, W Von Wendorff, N Nguyen, P Diehl, R Meertens, R Johannson. Automated driving safety—the art of conscious risk taking—minimum lateral distances to pedestrians. 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE); 2021 Feb 1-5, IEEE, Grenoble, France. Piscataway (2021), pp. 1466-1471

[226]

Rafrafi M, Bourdeaud’Huy T, Risk apportionment methodology based on functional analysis. In: Proceedings of the Multiconference on “Computational Engineering in Systems Applications”; 2006 Oct 4-6; Beijing, China. Piscataway: IEEE; 1103-9.

[227]

H Langdalen, EB Abrahamsen, JT Selvik. On the importance of systems thinking when using the ALARP principle for risk management. Reliab Eng Syst Saf, 204 (2020), p. 107222

[228]

X Tang, K Yang, H Wang, J Wu, Y Qin, W Yu, et al.. Prediction-uncertainty-aware decision-making for autonomous vehicles. IEEE Trans. Intell Veh, 7 (4) (2022), pp. 849-862

[229]

K Yang, X Tang, S Qiu, S Jin, Z Wei, H Wang. Towards robust decision-making for autonomous driving on highway. IEEE Trans Vehicular Technol, 72 (9) (2023), pp. 11251-11263

[230]

Peng L, Li B, Yu W, Yang K, Shao W, Wang H. SOTIF entropy: online SOTIF risk quantification and mitigation for autonomous driving. IEEE Trans Intell Transp Syst. In press.

[231]

L. Peng, H. Wang, J. Li. Uncertainty evaluation of object detection algorithms for autonomous vehicles. Automot Innov, 4 (2021), pp. 241-252

[232]

J Liu, H Wang, Z Cao, W Yu, C Zhao, D Zhao, et al.. Semantic traffic law adaptive decision-making for self-driving vehicles. IEEE Trans Intell Transp Syst, 24 (12) (2023), pp. 14858-14872

[233]

S Li, J Zhang, S Wang, P Li, Y Liao. Ethical and legal dilemma of autonomous vehicles: study on driving decision-making model under the emergency situations of red light-running behaviors. Electronics, 7 (10) (2018), p. 264

[234]

H Wang, A Khajepour, D Cao, T Liu. Ethical decision making in autonomous vehicles: challenges and research progress. IEEE Intell. Transport Syst Mag, 14 (2022), pp. 6-17

[235]

E Medvet, A Bartoli, J Talamini. Road traffic rules synthesis using grammatical evolution. G Squillero, K Sim (Eds.), Applications of Evolutionary Computation, Springer, Berlin (2017), pp. 173-188

[236]

J Talamini, A Bartoli, A De Lorenzo, E Medvet. On the impact of the rules on autonomous drive learning. Appl Sci, 10 (7) (2020), p. 2394