Journal Home Online First Current Issue Archive For Authors Journal Information 中文版

Engineering >> 2022, Volume 18, Issue 11 doi: 10.1016/j.eng.2022.04.025

A Local Quadratic Embedding Learning Algorithm and Applications for Soft Sensing

Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai 200237, China

Received: 2019-12-30 Revised: 2021-01-25 Accepted: 2022-04-29 Available online: 2022-09-28

Next Previous

Abstract

Inspired by the tremendous achievements of meta-learning in various fields, this paper proposes the local quadratic embedding learning (LQEL) algorithm for regression problems based on metric learning and neural networks (NNs). First, Mahalanobis metric learning is improved by optimizing the global consistency of the metrics between instances in the input and output space. Then, we further prove that the improved metric learning problem is equivalent to a convex programming problem by relaxing the constraints. Based on the hypothesis of local quadratic interpolation, the algorithm introduces two lightweight NNs; one is used to learn the coefficient matrix in the local quadratic model, and the other is implemented for weight assignment for the prediction results obtained from different local neighbors. Finally, the two sub-models are embedded in a unified regression framework, and the parameters are learned by means of a stochastic gradient descent (SGD) algorithm. The proposed algorithm can make full use of the information implied in target labels to find more reliable reference instances. Moreover, it prevents the model degradation caused by sensor drift and unmeasurable variables by modeling variable differences with the LQEL algorithm. Simulation results on multiple benchmark datasets and two practical industrial applications show that the proposed method outperforms several popular regression methods.

Figures

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

References

[ 1 ] Bao YY, Zhu YM, Du WL, Zhong WM, Qian F. A distributed PCA-TSS based soft sensor for raw meal fineness in VRM system. Control Eng Pract 2019;90:38–49. link1

[ 2 ] Zhong WM, Jiang C, Peng X, Li Z, Qian F. Online quality prediction of industrial terephthalic acid hydropurification process using modified regularized slowfeature analysis. Ind Eng Chem Res 2018;57(29):9604–14. link1

[ 3 ] Lu B, Chiang L. Semi-supervised online soft sensor maintenance experiences in the chemical industry. J Process Contr 2018;67:23–34. link1

[ 4 ] Li YG, Gui WH, Yang CH, Xie YF. Soft sensor and expert control for blending and digestion process in alumina metallurgical industry. J Process Contr 2013;23 (7):1012–21. link1

[ 5 ] Song YC, Zhou H, Wang PF, Yang MJ. Prediction of clathrate hydrate phase equilibria using gradient boosted regression trees and deep neural networks. J Chem Thermodyn 2019;135:86–96. link1

[ 6 ] Touzani S, Granderson J, Fernandes S. Gradient boosting machine for modeling the energy consumption of commercial buildings. Energ Build 2018;158:1533–43. link1

[ 7 ] Fernández-Delgado M, Sirsat MS, Cernadas E, Alawadi S, Barro S, FebreroBande M. An extensive experimental survey of regression methods. Neural Netw 2019;111:11–34. link1

[ 8 ] Yu J. Multiway Gaussian mixture model based adaptive kernel partial least squares regression method for soft sensor estimation and reliable quality prediction of nonlinear multiphase batch processes. Ind Eng Chem Res 2012;51 (40):13227–37. link1

[ 9 ] Yuan XF, Ge ZQ, Song ZH. Locally weighted kernel principal component regression model for soft sensing of nonlinear time-variant processes. Ind Eng Chem Res 2014;53(35):13736–49. link1

[10] Gou JP, Ma HX, Ou WH, Zeng SN, Rao YB, Yang HB. A generalized mean distance-based k-nearest neighbor classifier. Expert Syst Appl 2019;115: 356–72. link1

[11] Juez-Gil M, Erdakov IN, Bustillo A, Pimenov DY. A regression-tree multilayerperceptron hybrid strategy for the prediction of ore crushing-plate lifetimes. J Adv Res 2019;18:173–84. link1

[12] Suarez A, Lutsko JF. Globally optimal fuzzy decision trees for classification and regression. IEEE Trans Pattern Anal Mach Intell 1999;21(12):1297–311. link1

[13] Huang GB, Zhou HM, Ding XJ, Zhang R. Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern B Cybern 2012;42(2):513–29. link1

[14] Vilela LFS, Leme RC, Pinheiro CAM, Carpinteiro OAS. Forecasting financial series using clustering methods and support vector regression. Artif Intell Rev 2019;52(2):743–73. link1

[15] Paul A, Mukherjee DP. Reinforced quasi-random forest. Pattern Recognit 2019;94:13–24. link1

[16] Rodriguez-Galiano VF, Ghimire B, Rogan J, Chica-Olmo M, Rigol-Sanchez JP. An assessment of the effectiveness of a random forest classifier for land-cover classification. ISPRS J Photogramm Remote Sens 2012;67:93–104. link1

[17] Freund Y. Boosting a weak learning algorithm by majority. Inf Comput 1995;121(2):256–85. link1

[18] Chen TQ, Guestrin C. XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2016 Aug 13–17; San Francisco, CA, USA; 2016. p. 785–94.

[19] Dreiseitl S, Ohno-Machado L. Logistic regression and artificial neural network classification models: a methodology review. J Biomed Inform 2002;35(5– 6):352–9. link1

[20] Zhou ZH, Wu JX, Tang W. Ensembling neural networks: many could be better than all. Artif Intell 2002;137(1–2):239–63. link1

[21] Liu WB, Wang ZD, Liu XH, Zeng NY, Liu YR, Alsaadi FE. A survey of deep neural network architectures and their applications. Neurocomputing 2017;234:11–26. link1

[22] Martos G,Muñoz A, González J. On the generalization of the Mahalanobis distance. In: Ruiz-Shulcloper J, Sanniti di Baja G, editors. Progress in pattern recognition, image analysis, computer vision, and applications. Berlin: Springer; 2013. p. 125–32. link1

[23] Atkeson CG, Moore AW, Schaal S. Locally weighted learning. Artif Intell Rev 1997;11:11–73. link1

[24] Zhang JJ. Identification of moving loads using a local linear embedding algorithm. J Vib Control 2019;25(11):1780–90. link1

[25] Loia V, Tomasiello S, Vaccaro A, Gao JW. Using local learning with fuzzy transform: application to short term forecasting problems. Fuzzy Optim Decis Making 2020;19(1):13–32. link1

[26] Weinberger KQ, Saul LK. Distance metric learning for large margin nearest neighbor classification. J Mach Learn Res 2009;10:207–44. link1

[27] Nguyen B, Morell C, De Baets B. Large-scale distance metric learning for knearest neighbors regression. Neurocomputing 2016;214:805–14. link1

[28] Xing EP, Ng AY, Jordan MI, Russell S. Distance metric learning, with application to clustering with side-information. In: Becker S, Thrun S, Obermayer K, editors. Advances in neural information processing systems 15: proceedings of the 2002 conference. Cambridge: A Bradford Book; 2003. p. 521–8.

[29] Duan YQ, Lu JW, Zheng WZ, Zhou J. Deep adversarial metric learning. IEEE Trans Image Process 2020;29:2037–51. link1

[30] Song HO, Xiang Y, Jegelka S, Savarese S. Deep metric learning via lifted structured feature embedding. In: Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition; 2016 Jun 26–Jul 1; Las Vegas, NV, USA; 2016. p. 4004–12.

[31] Cui Y, Zhou F, Lin YQ, Belongie S. Fine-grained categorization and dataset bootstrapping using deep metric learning with humans in the loop. In: Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition; 2016 Jun 26–Jul 1; Las Vegas, NV, USA; 2016. p. 1153–62.

[32] ALzubi JA, Bharathikannan B, Tanwar S, Manikandan R, Khanna A, Thaventhiran C. Boosted neural network ensemble classification for lung cancer disease diagnosis. Appl Soft Comput 2019;80:579–91. link1

[33] Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. 2015. arXiv:1502.03167.

[34] Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR. Improving neural networks by preventing co-adaptation of feature detectors. 2012. arXiv:1207.0580.

[35] Kay S. Can detectability be improved by adding noise? IEEE Signal Process Lett 2000;7(1):8–10. link1

[36] Boyd V, Vandenberghe L, Faybusovich L. Convex optimization. IEEE Trans Automat Contr 2006;51(11):1859. link1

[37] Cortez P, Morais A. A data mining approach to predict forest fires using meteorological data. In: Neves JM, Santos MF, Machado JM, editors. New trends in artificial intelligence: proceedings of the 13th Portuguese Conference on Artificial Intelligence; 2007 Dec 3–7; Guimarães, Portugal; 2007. p. 512–23. French.

[38] Cortez P, Cerdeira A, Almeida F, Matos T, Reis J. Modeling wine preferences by data mining from physicochemical properties. Decis Support Syst 2009;47 (4):547–53. link1

[39] De Vito S, Massera E, Piga M, Martinotto L, Di Francia G. On field calibration of an electronic nose for benzene estimation in an urban pollution monitoring scenario. Sens Actuators B Chem 2008;129(2):750–7. link1

[40] Ke GL, Meng Q, Finley T, Wang TF, Chen W, Ma WD, et al. LightGBM: a highly efficient gradient boosting decision tree. In: Guyon I, Von Luxburg U, Bengio S, Wallach H, Fergus R, Vishwanathan S, et al., editors. Proceedings of the 31st Annual Conference on Neural Information Processing Systems; 2017 Dec 4–9; Long Beach, CA, USA; 2017.

[41] Guo HF, Tang RM, Ye YM, Li ZG, He XQ. DeepFM: a factorization-machine based neural network for CTR prediction. In: Sierra C, editor. Proceedings of the 26th International Joint Conference on Artificial Intelligence; 2017 Aug 19–25; Melbourne, VIC, Australia; 2017. p. 1725–31.

[42] Zhang L, Shen WC, Huang JH, Li SJ, Pan G. Field-aware neural factorization machine for click-through rate prediction. IEEE Access 2019;7:75032–40. link1

[43] Huang JY, Zhang X, Fang BX. CoStock: a DeepFM model for stock market prediction with attentional embeddings. In: Proceedings of 2019 IEEE International Conference on Big Data; 2019 Dec 9–12; Los Angeles, CA, USA. New York City: IEEE; 2019. p. 5522–31.

Related Research